{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb)'s YOLO V2 loss\n",
    "<img src=\"https://farm8.staticflickr.com/7904/45592720625_821897e898_b.jpg\" width=\"1024\" height=\"570\" alt=\"YOLO v2 loss funciton\"></a>\n",
    "\n",
    "\n",
    "This is the fourth blog post of [Object Detection with YOLO blog series](https://fairyonice.github.io/tag/object-detection-using-yolov2-on-pascal-voc2012-series.html). \n",
    "This blog discusses the YOLO's loss funciton. This will be the most intense blog post in [Object Detection with YOLO blog series](https://fairyonice.github.io/tag/object-detection-using-yolov2-on-pascal-voc2012-series.html). as loss function of YOLO is quite complex. So please get excited! \n",
    "For demonstration of the code, I will agian use PASCAL VOC2012 data. \n",
    "This blog assumes that the readers have read the previous blog posts - [Part 1](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html), [Part 2](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html) and [Part 3](https://fairyonice.github.io/Part_3_Object_Detection_with_Yolo_using_VOC_2012_data_model.html).\n",
    "\n",
    "\n",
    "## Andrew Ng's YOLO lecture\n",
    "- [Neural Networks - Bounding Box Predictions](https://www.youtube.com/watch?v=gKreZOUi-O0&t=0s&index=7&list=PL_IHmaMAvkVxdDOBRg2CbcJBq9SY7ZUvs)\n",
    "- [C4W3L06 Intersection Over Union](https://www.youtube.com/watch?v=ANIzQ5G-XPE&t=7s)\n",
    "- [C4W3L07 Nonmax Suppression](https://www.youtube.com/watch?v=VAo84c1hQX8&t=192s)\n",
    "- [C4W3L08 Anchor Boxes](https://www.youtube.com/watch?v=RTlwl2bv0Tg&t=28s)\n",
    "- [C4W3L09 YOLO Algorithm](https://www.youtube.com/watch?v=9s_FpMpdYW8&t=34s)\n",
    "\n",
    "## Reference\n",
    "- [You Only Look Once:Unified, Real-Time Object Detection](https://arxiv.org/pdf/1506.02640.pdf) \n",
    "\n",
    "- [YOLO9000:Better, Faster, Stronger](https://arxiv.org/pdf/1612.08242.pdf)\n",
    " \n",
    "- [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2)\n",
    "\n",
    "## Reference in my blog\n",
    "- [Part 1 Object Detection using YOLOv2 on Pascal VOC2012 - anchor box clustering](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html)\n",
    "- [Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html)\n",
    "- [Part 3 Object Detection using YOLOv2 on Pascal VOC2012 - model](https://fairyonice.github.io/Part_3_Object_Detection_with_Yolo_using_VOC_2012_data_model.html)\n",
    "- [Part 4 Object Detection using YOLOv2 on Pascal VOC2012 - loss](https://fairyonice.github.io/Part_4_Object_Detection_with_Yolo_using_VOC_2012_data_loss.html)\n",
    "- [Part 5 Object Detection using YOLOv2 on Pascal VOC2012 - training](https://fairyonice.github.io/Part_5_Object_Detection_with_Yolo_using_VOC_2012_data_training.html)\n",
    "- [Part 6 Object Detection using YOLOv2 on Pascal VOC 2012 data - inference on image](https://fairyonice.github.io/Part_6_Object_Detection_with_Yolo_using_VOC_2012_data_inference_image.html)\n",
    "- [Part 7 Object Detection using YOLOv2 on Pascal VOC 2012 data - inference on video](https://fairyonice.github.io/Part_7_Object_Detection_with_Yolo_using_VOC_2012_data_inference_video.html)\n",
    "\n",
    "## My GitHub repository \n",
    "This repository contains all the ipython notebooks in this blog series and the funcitons (See backend.py). \n",
    "- [FairyOnIce/ObjectDetectionYolo](https://github.com/FairyOnIce/ObjectDetectionYolo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yumikondo/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.6.3 |Anaconda, Inc.| (default, Oct  6 2017, 12:04:38) \n",
      "[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os, sys\n",
    "import tensorflow as tf\n",
    "print(sys.version)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define anchor box\n",
    "<code>ANCHORS</code> defines the number of anchor boxes and the shape of each anchor box.\n",
    "The choice of the anchor box specialization is already discussed in [Part 1 Object Detection using YOLOv2 on Pascal VOC2012 - anchor box clustering](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html). \n",
    "\n",
    "Based on the K-means analysis in the previous blog post, I will select 4 anchor boxes of following width and height. The width and heights are rescaled in the grid cell scale (Assuming that the number of grid size is 13 by 13.) See [Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html) to learn how I rescal the anchor box shapes into the grid cell scale.\n",
    "\n",
    "Here I choose 4 anchor boxes. With 13 by 13 grids, every frame gets 4 x 13 x 13 = 676 bouding box predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ANCHORS = np.array([1.07709888,  1.78171903,  # anchor box 1, width , height\n",
    "                    2.71054693,  5.12469308,  # anchor box 2, width,  height\n",
    "                   10.47181473, 10.09646365,  # anchor box 3, width,  height\n",
    "                    5.48531347,  8.11011331]) # anchor box 4, width,  height"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Label vector containing 20 object classe names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LABELS = ['aeroplane',  'bicycle', 'bird',  'boat',      'bottle', \n",
    "          'bus',        'car',      'cat',  'chair',     'cow',\n",
    "          'diningtable','dog',    'horse',  'motorbike', 'person',\n",
    "          'pottedplant','sheep',  'sofa',   'train',   'tvmonitor']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# YOLOv2's loss function\n",
    "Before discussing the loss function, I will read in VOC2012 data, and call the batch generator in order to use the example batch <code> [x_batch,b_batch],y_batch</code> to demonstrate the usage of the loss funciton. \n",
    "### Read images and annotations into memory\n",
    "Use the pre-processing code for parsing annotation at [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2).\n",
    "This <code>parse_annoation</code> function is already used in [Part 1 Object Detection using YOLOv2 on Pascal VOC2012 - anchor box clustering](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html) and saved in my python script. \n",
    "This script can be downloaded at [my Github repository, FairyOnIce/ObjectDetectionYolo/backend](https://github.com/FairyOnIce/ObjectDetectionYolo/blob/master/backend.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "N train = 17125\n"
     ]
    }
   ],
   "source": [
    "### The location where the VOC2012 data is saved.\n",
    "train_image_folder = \"../ObjectDetectionRCNN/VOCdevkit/VOC2012/JPEGImages/\"\n",
    "train_annot_folder = \"../ObjectDetectionRCNN/VOCdevkit/VOC2012/Annotations/\"\n",
    "\n",
    "np.random.seed(1)\n",
    "from backend import parse_annotation\n",
    "train_image, seen_train_labels = parse_annotation(train_annot_folder,\n",
    "                                                  train_image_folder, \n",
    "                                                  labels=LABELS)\n",
    "print(\"N train = {}\".format(len(train_image)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate batch generator object\n",
    "<code>SimpleBatchGenerator</code> is discussed and used in \n",
    "[Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html).\n",
    "This script can be downloaded at [my Github repository, FairyOnIce/ObjectDetectionYolo/backend](https://github.com/FairyOnIce/ObjectDetectionYolo/blob/master/backend.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from backend import SimpleBatchGenerator\n",
    "\n",
    "BATCH_SIZE        = 500\n",
    "IMAGE_H, IMAGE_W  = 416, 416\n",
    "GRID_H,  GRID_W   = 13 , 13\n",
    "TRUE_BOX_BUFFER   = 50\n",
    "BOX               = int(len(ANCHORS)/2)\n",
    "# CLASS             = len(LABELS)\n",
    "\n",
    "generator_config = {\n",
    "    'IMAGE_H'         : IMAGE_H, \n",
    "    'IMAGE_W'         : IMAGE_W,\n",
    "    'GRID_H'          : GRID_H,  \n",
    "    'GRID_W'          : GRID_W,\n",
    "    'LABELS'          : LABELS,\n",
    "    'ANCHORS'         : ANCHORS,\n",
    "    'BATCH_SIZE'      : BATCH_SIZE,\n",
    "    'TRUE_BOX_BUFFER' : TRUE_BOX_BUFFER,\n",
    "}\n",
    "\n",
    "\n",
    "def normalize(image):\n",
    "    return image / 255.\n",
    "train_batch_generator = SimpleBatchGenerator(train_image, generator_config,\n",
    "                                             norm=normalize, shuffle=True)\n",
    "\n",
    "[x_batch,b_batch],y_batch = train_batch_generator.__getitem__(idx=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating YOLOv2's loss function\n",
    "I have seen a lot of online blog posts about YOLO v1 loss function.\n",
    "For example, at [Understanding YOLO](https://hackernoon.com/understanding-yolo-f5a74bbc7967).\n",
    "However, most of these posts discusses the loss function of Yolo v1 which must be different from Yolo v2.\n",
    "The two losses are different and the lack of explicit formula in the Yolo v2 loss paper rises some confusion, for example at [What is YOLOv2 Loss Function - Google Groups](https://groups.google.com/forum/#!topic/darknet/TJ4dN9R4iJk).\n",
    "\n",
    "\n",
    "The YOLO v1 is difined in \n",
    "[You Only Look Once:Unified, Real-Time Object Detection](https://arxiv.org/pdf/1506.02640.pdf) as:\n",
    "\n",
    "### YOLO V1 loss\n",
    "\n",
    "$$\\begin{array}{rl}\n",
    "&\\lambda_\\textbf{coord}\n",
    "\\sum_{i = 0}^{S^2}\n",
    "    \\sum_{j = 0}^{B}\n",
    "     L_{ij}^{\\text{obj}}\n",
    "            \\left[\n",
    "            \\left(\n",
    "                x_i - \\hat{x}_i\n",
    "            \\right)^2 +\n",
    "            \\left(\n",
    "                y_i - \\hat{y}_i\n",
    "            \\right)^2\n",
    "            \\right]\n",
    "\\\\\n",
    "&+ \\lambda_\\textbf{coord} \n",
    "\\sum_{i = 0}^{S^2}\n",
    "    \\sum_{j = 0}^{B}\n",
    "         L_{ij}^{\\text{obj}}\n",
    "         \\left[\n",
    "        \\left(\n",
    "            \\sqrt{w_i} - \\sqrt{\\hat{w}_i}\n",
    "        \\right)^2 +\n",
    "        \\left(\n",
    "            \\sqrt{h_i} - \\sqrt{\\hat{h}_i}\n",
    "        \\right)^2\n",
    "        \\right]\n",
    "\\\\\n",
    "&+ \\sum_{i = 0}^{S^2}\n",
    "    \\sum_{j = 0}^{B}\n",
    "        L_{ij}^{\\text{obj}}\n",
    "        \\left(\n",
    "            C_i - \\hat{C}_i\n",
    "        \\right)^2\n",
    "\\\\\n",
    "&+ \\lambda_\\textrm{noobj}\n",
    "\\sum_{i = 0}^{S^2}\n",
    "    \\sum_{j = 0}^{B}\n",
    "    L_{ij}^{\\text{noobj}}\n",
    "        \\left(\n",
    "            C_i - \\hat{C}_i\n",
    "        \\right)^2\n",
    "\\\\\n",
    "&+ \\sum_{i = 0}^{S^2}\n",
    "L_i^{\\text{obj}}\n",
    "    \\sum_{c \\in \\textrm{classes}}\n",
    "        \\left(\n",
    "            p_i(c) - \\hat{p}_i(c)\n",
    "        \\right)^2\n",
    "\\end{array}$$\n",
    "\n",
    "YOLOv2 paper expalins the difference in architecture from YOLOv1 as follows:\n",
    "\n",
    "<blockquote>\n",
    "We remove the fully connected layers from YOLO(v1) and use anchor boxes to predict bounding boxes...\n",
    "When we move to anchor boxes we also decouple the class prediction mechanism from the spatial location and instead predict class and objectness for every anchorbox. \n",
    "</blockquote>\n",
    "\n",
    "This means that the confidence probability $p_i(c)$ above should depend not only on $i$ and $c$ but also an anchor box index, say $j$. Therefore, the loss needs to be different from above. Unfortunately, YOLOv2 paper does not explicitly state its loss function. So rather than making a guess based on the paper, I will try to understand the loss function that [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb) defines for his YOLOv2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb)'s YOLO V2 loss\n",
    "\n",
    "In [Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html), I showed that YOLO's output is encoded into <code>y_batch</code>.\n",
    "\n",
    "Here I revisit YOLO's ground truth output encoding with mathematical notations.\n",
    "\n",
    "\n",
    "<code>\n",
    "y_batch\n",
    "</code>\n",
    "\n",
    "The numpy array of shape  <code>(BATCH_SIZE, GRID_H, GRID_W, BOX, 4 + 1 + N classes) </code>.\n",
    "\n",
    "BOX = The number of anchor boxes.\n",
    "\n",
    "In the following notation, the grid cell index can be defined by two index (<code>igrid_h</code>,<code>igrid_w</code>) or by a single index $i$; (<code>igrid_h</code>,<code>igrid_w</code>) $\\leftrightarrow i$\n",
    "\n",
    "- <code>y_batch[iframe,igrid_h,igrid_w,j,:4]</code> contains <code>(center_x,center_y,center_w,center_h)</code> of <code>j</code>th anchor at <code>grid cell=(igrid_h,igrid_w)</code> = $i$, \n",
    "    if the object exists in this (grid cell, anchor) pair, else they simply contain 0.\n",
    "    \n",
    "    <code>y_batch[iframe,igrid_h,igrid_w,j,:4]</code>$= (x_{i,\\texttt{j}},y_{i,\\texttt{j}},w_{i,\\texttt{j}},h_{i,\\texttt{j}})$\n",
    "    \n",
    "\n",
    "- <code>y_batch[iframe,igrid_h,igrid_w,j,4]</code> \n",
    "    contains 1 if the object exists in this (grid cell, anchor) pair, else it contains 0.\n",
    "    \n",
    "     <code>y_batch[iframe,igrid_h,igrid_w,j,4]</code> = $ C_{i,\\texttt{j}}$\n",
    "\n",
    "\n",
    "- <code>y_batch[iframe,igrid_h,igrid_w,j,5 + iclass]</code> contains 1 if the <code>iclass</code>th class object exists in this (grid cell, anchor) pair, else it contains 0.\n",
    "\n",
    "     <code>y_batch[iframe,igrid_h,igrid_w,j,5:]</code> $ = (p^1_{i,\\texttt{j}},p^2_{i,\\texttt{j}},p^3_{i,\\texttt{j}},\\cdots,p^{\\textrm{Nclass}}_{i,\\texttt{j}})$\n",
    "\n",
    "The loss function of YOLO treats each set of elements \n",
    "<code>(center_x,center_y,center_w,center_h)</code>, \n",
    "<code>C</code>, and \n",
    "<code>(p_1,p_2,...p_Nclass)</code> in <code>y_batch[iframe,igrid_h,igrid_w,:]</code> differently. So let's understand each term of the loss one by one.\n",
    "\n",
    "\n",
    "Then the loss corresponding to (grid cell, anchor box) pair = ($i,j$) is calculated as:\n",
    "\n",
    "$$\\small\n",
    "%%%\n",
    "\\begin{array}{rl}\\small\n",
    "\\textrm{loss}_{i,j} &= \\textrm{loss}_{i,j}^{xywh} +  \\textrm{loss}_{i,j}^p + \\textrm{loss}_{i,j}^c \\\\\n",
    "%%%\n",
    "\\textrm{loss}_{i,j}^{xywh}&=\n",
    "\\frac{\\lambda_{\\textrm{coord}}}{N_{L^{obj}}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{obj}}\n",
    "\\big[\n",
    "\\left(x_{i,j}-\\hat{x}_{i,j}\\right)^2 + \n",
    "\\left(y_{i,j}-\\hat{y}_{i,j}\\right)^2 +\\\\\n",
    "&\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\left(\\sqrt{w}_{i,j}-\\sqrt{\\hat{w}}_{i,j}\\right)^2 +\n",
    "\\left(\\sqrt{h}_{i,j}-\\sqrt{\\hat{h}}_{i,j}\\right)^2 \n",
    "\\big]\\\\\n",
    "%%%\n",
    "\\textrm{loss}_{i,j}^p&=\n",
    "-\\frac{\\lambda_{\\text{class}}}{N_{L^{obj}}}\n",
    "\\sum_{i=0}^{S^2} \\sum_{j=0}^B L_{i,j}^{\\text{obj}}\n",
    "\\sum_{c \\in \\text{class}}  p_{i,j}^c \\text{log}(\\hat{p}_{i,j}^c)\\\\\n",
    "%%%\n",
    "\\textrm{loss}_{i,j}^c &=\n",
    "\\frac{\\lambda_{\\text{obj}}}{N^{conf}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{obj}}\n",
    "\\left(\n",
    "IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i,j}} - \\widehat{C}_{i,j}\n",
    "\\right)^2\\\\\n",
    "&\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;+\n",
    "\\frac{\\lambda_{\\textrm{noobj}}}{N^{conf}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{noobj}}\n",
    "\\left(0 - \\widehat{C}_{i,j}\\right)\\\\\n",
    "\\end{array}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where: \n",
    "- $N_{L^{obj}} = \\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B L_{i,j}^{\\text{obj}}$\n",
    "- $N^{conf} =  \\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B L_{i,j}^{\\text{obj}} + L_{i,j}^{\\text{noobj}}(1-L_{i,j}^{\\text{obj}})$\n",
    "\n",
    "- $\\text{preduiction}_{i,j}=(\\hat{x}_{i,j},\\hat{y}_{i,j},\\hat{w}_{i,j},\\hat{h}_{i,j})$\n",
    "- $\\text{ground truth}_{i,j}=(x_{i,j},y_{i,j},w_{i,j},h_{i,j})$\n",
    "- $\\lambda_{\\text{coord}}$, $\\lambda_{\\text{class}}$ and $\\lambda_{\\text{noobj}}$ are scalars to weight each loss funciton \n",
    "\n",
    "Here, $L_{i,j}^{\\text{noobj}}$ and $L_{i,j}^{\\text{obj}}$ are 0/1 indicator function such that:\n",
    "$$\n",
    "\\begin{array}{rl}\n",
    "L_{i,j}^{\\text{obj}}\n",
    " &= \n",
    " \\begin{cases}\n",
    " 1 \n",
    "\\;\\;\\text{if} \\;\\;C_{i,j}=1\\\\\n",
    " 0\\;\\;\\text{else}\\\\\n",
    "\\end{cases}\\\\\n",
    "L_{i,j}^{\\text{noobj}}\n",
    "& = \n",
    "\\begin{cases}\n",
    " 1 \\;\\;\\text{if}\\;\\;\\text{max}_{i',j'}\n",
    " \\;\\;IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i',j'}} < 0.6 \\;\\text{and}\\; C_{i,j} = 0\\\\\n",
    " 0\\;\\;\\text{else}\\\\\n",
    "\\end{cases}\\\\\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "As the loss function seems complex, let's go over the  [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb)'s YOLO V2 loss function line by line. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Global hyperparameters necessary for the loss function\n",
    "                   -  true_boxes\n",
    "                   -  GRID_W\n",
    "                   -  GRID_H\n",
    "                   -  BATCH_SIZE\n",
    "                   -  ANCHORS\n",
    "                   -  LAMBDA_COORD\n",
    "                   -  LAMBDA_CLASS\n",
    "                   -  LAMBDA_NO_OBJECT \n",
    "                   -  LAMBDA_OBJECT\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LAMBDA_NO_OBJECT = 1.0\n",
    "LAMBDA_OBJECT    = 5.0\n",
    "LAMBDA_COORD     = 1.0\n",
    "LAMBDA_CLASS     = 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Adjust prediction output \n",
    "[experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb)'s YOLO V2 loss function starts off by rescaling the prediction output. \n",
    "Notice that the predicted y_batch output can take any real values as the conv_23 layer has a linear activation function. However, the prediction outputs should be rescaled in the following range:\n",
    "\n",
    "\n",
    "1. $\\hat{x}_{i,j}$ ranges between [<code>igrid_w,igrid_w+1</code>).\n",
    "2. $\\hat{y}_{i,j}$ ranges between [<code>igrid_h,igrid_h+1</code>)\n",
    "3. $\\hat{w}_{i,j}$ ranges between 0, GRID_W\n",
    "4. $\\hat{h}_{i,j}$ ranges between 0, GRID_H\n",
    "5. $\\widehat{C}_{i,j}$ ranges between 0 and 1. \n",
    "6. $\\widehat{p^c}_{i,j}$ ranges between 0 and 1\n",
    "\n",
    "The 1, 2, 3 and 4 are because the bounding boxes are in grid cell scale. See Bounding box encoding Section in \n",
    "  [Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_cell_grid(GRID_W,GRID_H,BATCH_SIZE,BOX): \n",
    "    '''\n",
    "    Helper function to assure that the bounding box x and y are in the grid cell scale\n",
    "    == output == \n",
    "    for any i=0,1..,batch size - 1\n",
    "    output[i,5,3,:,:] = array([[3., 5.],\n",
    "                               [3., 5.],\n",
    "                               [3., 5.]], dtype=float32)\n",
    "    '''\n",
    "    ## cell_x.shape = (1, 13, 13, 1, 1)\n",
    "    ## cell_x[:,i,j,:] = [[[j]]]\n",
    "    cell_x = tf.to_float(tf.reshape(tf.tile(tf.range(GRID_W), [GRID_H]), (1, GRID_H, GRID_W, 1, 1)))\n",
    "    ## cell_y.shape = (1, 13, 13, 1, 1)\n",
    "    ## cell_y[:,i,j,:] = [[[i]]]\n",
    "    cell_y = tf.transpose(cell_x, (0,2,1,3,4))\n",
    "    ## cell_gird.shape = (16, 13, 13, 5, 2)\n",
    "    ## for any n, k, i, j\n",
    "    ##    cell_grid[n, i, j, anchor, k] = j when k = 0\n",
    "    ## for any n, k, i, j\n",
    "    ##    cell_grid[n, i, j, anchor, k] = i when k = 1    \n",
    "    cell_grid = tf.tile(tf.concat([cell_x,cell_y], -1), [BATCH_SIZE, 1, 1, BOX, 1])\n",
    "    return(cell_grid) \n",
    "\n",
    "def adjust_scale_prediction(y_pred, cell_grid, ANCHORS):    \n",
    "    \"\"\"\n",
    "        Adjust prediction\n",
    "        \n",
    "        == input ==\n",
    "        \n",
    "        y_pred : takes any real values\n",
    "                 tensor of shape = (N batch, NGrid h, NGrid w, NAnchor, 4 + 1 + N class)\n",
    "        \n",
    "        ANCHORS : list containing width and height specializaiton of anchor box\n",
    "        == output ==\n",
    "        \n",
    "        pred_box_xy : shape = (N batch, N grid x, N grid y, N anchor, 2), contianing [center_y, center_x] rangining [0,0]x[grid_H-1,grid_W-1]\n",
    "          pred_box_xy[irow,igrid_h,igrid_w,ianchor,0] =  center_x\n",
    "          pred_box_xy[irow,igrid_h,igrid_w,ianchor,1] =  center_1\n",
    "          \n",
    "          calculation process:\n",
    "          tf.sigmoid(y_pred[...,:2]) : takes values between 0 and 1\n",
    "          tf.sigmoid(y_pred[...,:2]) + cell_grid : takes values between 0 and grid_W - 1 for x coordinate \n",
    "                                                   takes values between 0 and grid_H - 1 for y coordinate \n",
    "                                                   \n",
    "        pred_Box_wh : shape = (N batch, N grid h, N grid w, N anchor, 2), containing width and height, rangining [0,0]x[grid_H-1,grid_W-1]\n",
    "        \n",
    "        pred_box_conf : shape = (N batch, N grid h, N grid w, N anchor, 1), containing confidence to range between 0 and 1\n",
    "        \n",
    "        pred_box_class : shape = (N batch, N grid h, N grid w, N anchor, N class), containing \n",
    "    \"\"\"\n",
    "    BOX = int(len(ANCHORS)/2)\n",
    "    ## cell_grid is of the shape of \n",
    "    \n",
    "    ### adjust x and y  \n",
    "    # the bounding box bx and by are rescaled to range between 0 and 1 for given gird.\n",
    "    # Since there are BOX x BOX grids, we rescale each bx and by to range between 0 to BOX + 1\n",
    "    pred_box_xy = tf.sigmoid(y_pred[..., :2]) + cell_grid # bx, by\n",
    "    \n",
    "    ### adjust w and h\n",
    "    # exp to make width and height positive\n",
    "    # rescale each grid to make some anchor \"good\" at representing certain shape of bounding box \n",
    "    pred_box_wh = tf.exp(y_pred[..., 2:4]) * np.reshape(ANCHORS,[1,1,1,BOX,2]) # bw, bh\n",
    "\n",
    "    ### adjust confidence \n",
    "    pred_box_conf = tf.sigmoid(y_pred[..., 4])# prob bb\n",
    "\n",
    "    ### adjust class probabilities \n",
    "    pred_box_class = y_pred[..., 5:] # prC1, prC2, ..., prC20\n",
    "    \n",
    "    return(pred_box_xy,pred_box_wh,pred_box_conf,pred_box_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>adjust_scale_prediction</code>\n",
    "\n",
    "I generate the real valued y_pred before rescaling from Normal(mean=0,sd = const/(GRID_H*GRID_W)), as this will be my weight initializer for the layer 23 when const = 1. I will set const = 10 to make the distribution variance a bit larger. \n",
    "\n",
    "The bounding box parameters x, y, w, h and confidence are in the expected range. \n",
    "However, the class probabilities can take negative values. This is ok because the loss function later applies  softmax to <code>y_pred[:,:,:,:,5:]</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "prepare inputs\n",
      "y_pred before scaling = (500, 13, 13, 4, 25)\n",
      "******************************\n",
      "define tensor graph\n",
      "******************************\n",
      "ouput\n",
      "******************************\n",
      "\n",
      "pred_box_xy (500, 13, 13, 4, 2)\n",
      "  bounding box x at iGRID_W=00 MIN= 0.44, MAX= 0.55\n",
      "  bounding box x at iGRID_W=01 MIN= 1.44, MAX= 1.56\n",
      "  bounding box x at iGRID_W=02 MIN= 2.44, MAX= 2.56\n",
      "  bounding box x at iGRID_W=03 MIN= 3.43, MAX= 3.56\n",
      "  bounding box x at iGRID_W=04 MIN= 4.44, MAX= 4.56\n",
      "  bounding box x at iGRID_W=05 MIN= 5.44, MAX= 5.56\n",
      "  bounding box x at iGRID_W=06 MIN= 6.44, MAX= 6.56\n",
      "  bounding box x at iGRID_W=07 MIN= 7.44, MAX= 7.56\n",
      "  bounding box x at iGRID_W=08 MIN= 8.44, MAX= 8.56\n",
      "  bounding box x at iGRID_W=09 MIN= 9.44, MAX= 9.55\n",
      "  bounding box x at iGRID_W=10 MIN=10.44, MAX=10.56\n",
      "  bounding box x at iGRID_W=11 MIN=11.44, MAX=11.56\n",
      "  bounding box x at iGRID_W=12 MIN=12.44, MAX=12.56\n",
      "  bounding box y at iGRID_H=00 MIN= 0.44, MAX= 0.57\n",
      "  bounding box y at iGRID_H=01 MIN= 1.45, MAX= 1.56\n",
      "  bounding box y at iGRID_H=02 MIN= 2.44, MAX= 2.56\n",
      "  bounding box y at iGRID_H=03 MIN= 3.44, MAX= 3.57\n",
      "  bounding box y at iGRID_H=04 MIN= 4.44, MAX= 4.56\n",
      "  bounding box y at iGRID_H=05 MIN= 5.44, MAX= 5.56\n",
      "  bounding box y at iGRID_H=06 MIN= 6.44, MAX= 6.56\n",
      "  bounding box y at iGRID_H=07 MIN= 7.45, MAX= 7.55\n",
      "  bounding box y at iGRID_H=08 MIN= 8.44, MAX= 8.56\n",
      "  bounding box y at iGRID_H=09 MIN= 9.44, MAX= 9.56\n",
      "  bounding box y at iGRID_H=10 MIN=10.44, MAX=10.55\n",
      "  bounding box y at iGRID_H=11 MIN=11.44, MAX=11.57\n",
      "  bounding box y at iGRID_H=12 MIN=12.44, MAX=12.56\n",
      "\n",
      "pred_box_wh (500, 13, 13, 4, 2)\n",
      "  bounding box width  MIN= 0.81, MAX=13.47\n",
      "  bounding box height MIN= 1.37, MAX=12.87\n",
      "\n",
      "pred_box_conf (500, 13, 13, 4)\n",
      "  confidence  MIN= 0.43, MAX= 0.57\n",
      "\n",
      "pred_box_class (500, 13, 13, 4, 20)\n",
      "  class probability MIN=-0.30, MAX= 0.29\n"
     ]
    }
   ],
   "source": [
    "def print_min_max(vec,title):\n",
    "    print(\"{} MIN={:5.2f}, MAX={:5.2f}\".format(\n",
    "        title,np.min(vec),np.max(vec)))\n",
    "\n",
    "print(\"*\"*30)\n",
    "print(\"prepare inputs\")\n",
    "GRID_W = 13 \n",
    "GRID_H = 13 \n",
    "BOX    = int(len(ANCHORS)/2)\n",
    "CLASS = len(LABELS)\n",
    "size   = BATCH_SIZE*GRID_W*GRID_H*BOX*(4 + 1 + CLASS)\n",
    "y_pred = np.random.normal(size=size,scale = 10/(GRID_H*GRID_W)) \n",
    "y_pred = y_pred.reshape(BATCH_SIZE,GRID_H,GRID_W,BOX,4 + 1 + CLASS)\n",
    "print(\"y_pred before scaling = {}\".format(y_pred.shape))\n",
    "\n",
    "print(\"*\"*30)\n",
    "print(\"define tensor graph\")\n",
    "y_pred_tf = tf.constant(y_pred,dtype=\"float32\")\n",
    "cell_grid = get_cell_grid(GRID_W,GRID_H,BATCH_SIZE,BOX)\n",
    "(pred_box_xy_tf,   pred_box_wh_tf, \n",
    " pred_box_conf_tf, pred_box_class_tf) = adjust_scale_prediction(y_pred_tf, \n",
    "                                                                cell_grid, \n",
    "                                                                ANCHORS)\n",
    "print(\"*\"*30 + \"\\nouput\\n\" + \"*\"*30) \n",
    "with tf.Session() as sess:\n",
    "    (pred_box_xy,   pred_box_wh, \n",
    "     pred_box_conf, pred_box_class) = sess.run(\n",
    "        [pred_box_xy_tf,   pred_box_wh_tf,\n",
    "         pred_box_conf_tf, pred_box_class_tf])\n",
    "    \n",
    "print(\"\\npred_box_xy {}\".format(pred_box_xy.shape))           \n",
    "for igrid_w in range(pred_box_xy.shape[2]):\n",
    "        print_min_max(pred_box_xy[:,:,igrid_w,:,0].flatten(),\n",
    "                      \"  bounding box x at iGRID_W={:02.0f}\".format(igrid_w))\n",
    "for igrid_h in range(pred_box_xy.shape[1]):\n",
    "    print_min_max(pred_box_xy[:,igrid_h,:,:,1].flatten(),\n",
    "                  \"  bounding box y at iGRID_H={:02.0f}\".format(igrid_h)) \n",
    "    \n",
    "print(\"\\npred_box_wh {}\".format(pred_box_wh.shape))\n",
    "print_min_max(pred_box_wh[:,:,:,:,0].flatten(),\"  bounding box width \") \n",
    "print_min_max(pred_box_wh[:,:,:,:,1].flatten(),\"  bounding box height\") \n",
    "    \n",
    "print(\"\\npred_box_conf {}\".format(pred_box_conf.shape))\n",
    "print_min_max(pred_box_conf.flatten(),\"  confidence \") \n",
    "\n",
    "print(\"\\npred_box_class {}\".format(pred_box_class.shape))\n",
    "print_min_max(pred_box_class.flatten(),\"  class probability\") \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Extract ground truth output\n",
    "Extraction of the ground truth output is simpler than the extraction of the prediction output because the ground truth is already encoded in the correct scales. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract_ground_truth(y_true):    \n",
    "    true_box_xy    = y_true[..., 0:2] # bounding box x, y coordinate in grid cell scale \n",
    "    true_box_wh    = y_true[..., 2:4] # number of cells accross, horizontally and vertically\n",
    "    true_box_conf  = y_true[...,4]    # confidence \n",
    "    true_box_class = tf.argmax(y_true[..., 5:], -1)\n",
    "    return(true_box_xy, true_box_wh, true_box_conf, true_box_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>extract_ground_truth</code>\n",
    "The scales of the $C_{i,j}$ and ${p^c}_{i,j}$ are different from $\\widehat{C}_{i,j}$ and $\\widehat{p^c}_{i,j}$, as the their ground truths take 0/1 values: \n",
    "<table>\n",
    "  <tr>\n",
    "    <th></th>\n",
    "    <th>Estimate</th>\n",
    "    <th>Ground truth</th> \n",
    "    \n",
    "  </tr>\n",
    "  <tr>\n",
    "    <th>$C_{i,j}$</th>\n",
    "    <td>between 0 and 1 </td> \n",
    "    <td>1 (=an object exists) or 0</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>$p^c_{i,j}$</td>\n",
    "    <td>between 0 and 1</td> \n",
    "    <td>1 (=$c$th class object exists) or 0\n",
    "    </td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td></td>\n",
    "    <td></td> \n",
    "    <td>later transfered to class index  $c$</td>\n",
    "  </tr>  \n",
    "</table>\n",
    "\n",
    "\n",
    "\n",
    "When $C_{i,j}=1$, the scales of $x_{i,j}$, $y_{i,j}$, $w_{i,j}$, $h_{i,j}$ are the same as $\\hat{x}_{i,j}$, $\\hat{y}_{i,j}$, $\\hat{w}_{i,j}$, $\\hat{h}_{i,j}$.\n",
    "When $C_{i,j}=0$, then all bounding box parameters $x_{i,j}$, $y_{i,j}$, $w_{i,j}$, $h_{i,j}$ takes zero values as no object exists for the (grid_cell, anchor) pair.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input y_batch = (500, 13, 13, 4, 25)\n",
      "******************************\n",
      "ouput\n",
      "******************************\n",
      "\n",
      "true_box_xy (500, 13, 13, 4, 2)\n",
      "  bounding box x at iGRID_W=00 MIN= 0.17, MAX= 0.94\n",
      "  bounding box x at iGRID_W=01 MIN= 1.00, MAX= 1.97\n",
      "  bounding box x at iGRID_W=02 MIN= 2.02, MAX= 2.98\n",
      "  bounding box x at iGRID_W=03 MIN= 3.03, MAX= 3.97\n",
      "  bounding box x at iGRID_W=04 MIN= 4.02, MAX= 4.98\n",
      "  bounding box x at iGRID_W=05 MIN= 5.00, MAX= 5.98\n",
      "  bounding box x at iGRID_W=06 MIN= 6.00, MAX= 6.98\n",
      "  bounding box x at iGRID_W=07 MIN= 7.00, MAX= 7.98\n",
      "  bounding box x at iGRID_W=08 MIN= 8.02, MAX= 8.98\n",
      "  bounding box x at iGRID_W=09 MIN= 9.02, MAX= 9.98\n",
      "  bounding box x at iGRID_W=10 MIN=10.00, MAX=10.97\n",
      "  bounding box x at iGRID_W=11 MIN=11.00, MAX=11.98\n",
      "  bounding box x at iGRID_W=12 MIN=12.00, MAX=12.84\n",
      "  bounding box y at iGRID_H=00 MIN= 0.41, MAX= 0.92\n",
      "  bounding box y at iGRID_H=01 MIN= 1.02, MAX= 1.80\n",
      "  bounding box y at iGRID_H=02 MIN= 2.00, MAX= 2.97\n",
      "  bounding box y at iGRID_H=03 MIN= 3.03, MAX= 3.98\n",
      "  bounding box y at iGRID_H=04 MIN= 4.00, MAX= 4.98\n",
      "  bounding box y at iGRID_H=05 MIN= 5.00, MAX= 5.98\n",
      "  bounding box y at iGRID_H=06 MIN= 6.00, MAX= 6.98\n",
      "  bounding box y at iGRID_H=07 MIN= 7.00, MAX= 7.98\n",
      "  bounding box y at iGRID_H=08 MIN= 8.00, MAX= 8.98\n",
      "  bounding box y at iGRID_H=09 MIN= 9.00, MAX= 9.97\n",
      "  bounding box y at iGRID_H=10 MIN=10.00, MAX=10.97\n",
      "  bounding box y at iGRID_H=11 MIN=11.00, MAX=11.98\n",
      "  bounding box y at iGRID_H=12 MIN=12.02, MAX=12.41\n",
      "\n",
      "true_box_wh (500, 13, 13, 4, 2)\n",
      "  bounding box width  MIN= 0.00, MAX=13.00\n",
      "  bounding box height MIN= 0.00, MAX=13.00\n",
      "\n",
      "true_box_conf (500, 13, 13, 4)\n",
      "  confidence, unique value = [0. 1.]\n",
      "\n",
      "true_box_class (500, 13, 13, 4)\n",
      "  class index, unique value = [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]\n"
     ]
    }
   ],
   "source": [
    "# y_batch is the output of the simpleBatchGenerator.fit()\n",
    "print(\"Input y_batch = {}\".format(y_batch.shape))\n",
    "\n",
    "y_batch_tf = tf.constant(y_batch,dtype=\"float32\")\n",
    "(true_box_xy_tf, true_box_wh_tf, \n",
    " true_box_conf_tf, true_box_class_tf) = extract_ground_truth(y_batch_tf)\n",
    "\n",
    "print(\"*\"*30 + \"\\nouput\\n\" + \"*\"*30) \n",
    "\n",
    "with tf.Session() as sess:\n",
    "        (true_box_xy, true_box_wh, \n",
    "         true_box_conf, true_box_class) = sess.run(\n",
    "                    [true_box_xy_tf,   true_box_wh_tf, \n",
    "                     true_box_conf_tf, true_box_class_tf])\n",
    "\n",
    "print(\"\\ntrue_box_xy {}\".format(true_box_xy.shape))        \n",
    "for igrid_w in range(true_box_xy.shape[2]):\n",
    "    vec  = true_box_xy[:,:,igrid_w,:,0].flatten()\n",
    "    pick = true_box_conf[:,:,igrid_w,:].flatten() == 1 ## only pick C_ij = 1\n",
    "    print_min_max(vec[pick],\"  bounding box x at iGRID_W={:02.0f}\".format(igrid_w))\n",
    "    \n",
    "for igrid_h in range(true_box_xy.shape[1]):\n",
    "    vec  = true_box_xy[:,igrid_h,:,:,1].flatten()\n",
    "    pick = true_box_conf[:,igrid_h,:,:].flatten() == 1 ## only pick C_ij = 1\n",
    "    print_min_max(vec[pick],\"  bounding box y at iGRID_H={:02.0f}\".format(igrid_h)) \n",
    "    \n",
    "print(\"\\ntrue_box_wh {}\".format(true_box_wh.shape))\n",
    "print_min_max(true_box_wh[:,:,:,:,0].flatten(),\"  bounding box width \") \n",
    "print_min_max(true_box_wh[:,:,:,:,1].flatten(),\"  bounding box height\") \n",
    "    \n",
    "print(\"\\ntrue_box_conf {}\".format(true_box_conf.shape))\n",
    "print(\"  confidence, unique value = {}\".format(np.unique(true_box_conf.flatten()))) \n",
    "\n",
    "print(\"\\ntrue_box_class {}\".format(true_box_class.shape))\n",
    "print(\"  class index, unique value = {}\".format(np.unique(true_box_class.flatten())) )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: calculate $\\text{loss}_{i,j}^{xywh}$\n",
    "Now we are ready to calculate the loss specific to the bounding box parameters. \n",
    "\n",
    "\\begin{array}{rl}\n",
    "\\textrm{loss}_{i,j}^{xywh}&=\n",
    "\\frac{1}{N_{L^{obj}}}\n",
    "\\lambda_{\\textrm{coord}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{obj}}\n",
    "\\big[\n",
    "\\left(x_{i,j}-\\hat{x}_{i,j}\\right)^2 + \n",
    "\\left(y_{i,j}-\\hat{y}_{i,j}\\right)^2 +\\\\\n",
    "&\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\left(\\sqrt{w}_{i,j}-\\sqrt{\\hat{w}}_{i,j}\\right)^2 +\n",
    "\\left(\\sqrt{h}_{i,j}-\\sqrt{\\hat{h}}_{i,j}\\right)^2 \n",
    "\\big]\n",
    "\\end{array}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_loss_xywh(true_box_conf,\n",
    "                   COORD_SCALE,\n",
    "                   true_box_xy, pred_box_xy,true_box_wh,pred_box_wh):  \n",
    "    '''\n",
    "    coord_mask:      np.array of shape (Nbatch, Ngrid h, N grid w, N anchor, 1)\n",
    "                     lambda_{coord} L_{i,j}^{obj}     \n",
    "                         \n",
    "    '''\n",
    "    \n",
    "    # lambda_{coord} L_{i,j}^{obj} \n",
    "    # np.array of shape (Nbatch, Ngrid h, N grid w, N anchor, 1)\n",
    "    coord_mask  = tf.expand_dims(true_box_conf, axis=-1) * COORD_SCALE \n",
    "    nb_coord_box = tf.reduce_sum(tf.to_float(coord_mask > 0.0))\n",
    "    loss_xy      = tf.reduce_sum(tf.square(true_box_xy-pred_box_xy) * coord_mask) / (nb_coord_box + 1e-6) / 2.\n",
    "    loss_wh      = tf.reduce_sum(tf.square(true_box_wh-pred_box_wh) * coord_mask) / (nb_coord_box + 1e-6) / 2.\n",
    "    return(loss_xy + loss_wh, coord_mask)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>calc_loss_xywh</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "ouput\n",
      "******************************\n",
      "loss_xywh = 3.281\n"
     ]
    }
   ],
   "source": [
    "LAMBDA_COORD = 1\n",
    "loss_xywh_tf, coord_mask_tf  = calc_loss_xywh(true_box_conf_tf,LAMBDA_COORD,\n",
    "                                               true_box_xy_tf, pred_box_xy_tf,true_box_wh_tf,pred_box_wh_tf)\n",
    "\n",
    "print(\"*\"*30 + \"\\nouput\\n\" + \"*\"*30) \n",
    "\n",
    "with tf.Session() as sess:\n",
    "        loss_xywh = sess.run([loss_xywh_tf])[0]\n",
    "print(\"loss_xywh = {:4.3f}\".format(loss_xywh))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: calculate $\\text{loss}_{i,j}^{p}$\n",
    "$$\n",
    "\\textrm{loss}_{i,j}^p=-\\frac{1}{N_{L^{obj}}}\n",
    "\\lambda_{\\text{class}}\\sum_{i=0}^{S^2} \\sum_{j=0}^B L_{i,j}^{\\text{obj}}\n",
    "\\sum_{c \\in \\text{class}}  p_{i,j}^c \\text{log}(\\hat{p}_{i,j}^c)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_loss_class(true_box_conf,CLASS_SCALE, true_box_class,pred_box_class):\n",
    "    '''\n",
    "    == input ==    \n",
    "    true_box_conf  : tensor of shape (N batch, N grid h, N grid w, N anchor)\n",
    "    true_box_class : tensor of shape (N batch, N grid h, N grid w, N anchor), containing class index\n",
    "    pred_box_class : tensor of shape (N batch, N grid h, N grid w, N anchor, N class)\n",
    "    CLASS_SCALE    : 1.0\n",
    "    \n",
    "    == output ==  \n",
    "    class_mask\n",
    "    if object exists in this (grid_cell, anchor) pair and the class object receive nonzero weight\n",
    "        class_mask[iframe,igridy,igridx,ianchor] = 1 \n",
    "    else: \n",
    "        0 \n",
    "    '''   \n",
    "    class_mask   = true_box_conf  * CLASS_SCALE ## L_{i,j}^obj * lambda_class\n",
    "    \n",
    "    nb_class_box = tf.reduce_sum(tf.to_float(class_mask > 0.0))\n",
    "    loss_class   = tf.nn.sparse_softmax_cross_entropy_with_logits(labels = true_box_class, \n",
    "                                                                  logits = pred_box_class)\n",
    "    loss_class   = tf.reduce_sum(loss_class * class_mask) / (nb_class_box + 1e-6)   \n",
    "    return(loss_class)\n",
    "\n",
    "\n",
    "#Example useage of tf.gather\n",
    "#indices = np.array([[0,0],\n",
    "#                    [1,0],\n",
    "#                    [0,1]])\n",
    "\n",
    "#arr  = tf.constant(indices)\n",
    "#temp = tf.gather(np.array([100,-20]), arr)\n",
    "#with tf.Session() as sess:\n",
    "#    t = sess.run(temp)\n",
    "#print(t)\n",
    "\n",
    "#[[100 100]\n",
    "# [-20 100]\n",
    "# [100 -20]]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>calc_loss_class</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "ouput\n",
      "******************************\n",
      "loss_class = 3.000\n"
     ]
    }
   ],
   "source": [
    "LAMBDA_CLASS   = 1\n",
    "loss_class_tf  = calc_loss_class(true_box_conf_tf,LAMBDA_CLASS,\n",
    "                                 true_box_class_tf,pred_box_class_tf)\n",
    "print(\"*\"*30 + \"\\nouput\\n\" + \"*\"*30) \n",
    "with tf.Session() as sess:\n",
    "        loss_class = sess.run(loss_class_tf)\n",
    "print(\"loss_class = {:4.3f}\".format(loss_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\textrm{loss}_{i,j}^c$\n",
    "The rest of calculation is dedicated to evaluate confidence loss $\\textrm{loss}_{i,j}^c$\n",
    "\n",
    "$$\n",
    "\\textrm{loss}_{i,j}^c =\\lambda_{\\text{obj}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{obj}}\n",
    "\\left(\n",
    "IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i,j}} - \\widehat{C}_{i,j}\n",
    "\\right)^2+\n",
    "\\lambda_{\\textrm{noobj}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{noobj}}\n",
    "\\left(0 - \\widehat{C}_{i,j}\\right)\n",
    "$$\n",
    "\n",
    "### Step 5, calculate $IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i,j}} $\n",
    "For each (grid cell, anchor) pair, compute IOU between ground truth bounding box and predicted bounding box.\n",
    "$IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i,j}} $ is 0 if $C_{i,j}=0$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_intersect_area(true_xy,true_wh,\n",
    "                       pred_xy,pred_wh):\n",
    "    '''\n",
    "    == INPUT ==\n",
    "    true_xy,pred_xy, true_wh and pred_wh must have the same shape length\n",
    "\n",
    "    p1 : pred_mins = (px1,py1)\n",
    "    p2 : pred_maxs = (px2,py2)\n",
    "    t1 : true_mins = (tx1,ty1) \n",
    "    t2 : true_maxs = (tx2,ty2) \n",
    "                 p1______________________ \n",
    "                 |      t1___________   |\n",
    "                 |       |           |  |\n",
    "                 |_______|___________|__|p2 \n",
    "                         |           |rmax\n",
    "                         |___________|\n",
    "                                      t2\n",
    "    intersect_mins : rmin = t1  = (tx1,ty1)\n",
    "    intersect_maxs : rmax = (rmaxx,rmaxy)\n",
    "    intersect_wh   : (rmaxx - tx1, rmaxy - ty1)\n",
    "        \n",
    "    '''\n",
    "    true_wh_half = true_wh / 2.\n",
    "    true_mins    = true_xy - true_wh_half\n",
    "    true_maxes   = true_xy + true_wh_half\n",
    "    \n",
    "    pred_wh_half = pred_wh / 2.\n",
    "    pred_mins    = pred_xy - pred_wh_half\n",
    "    pred_maxes   = pred_xy + pred_wh_half    \n",
    "    \n",
    "    intersect_mins  = tf.maximum(pred_mins,  true_mins)\n",
    "    intersect_maxes = tf.minimum(pred_maxes, true_maxes)\n",
    "    intersect_wh    = tf.maximum(intersect_maxes - intersect_mins, 0.)\n",
    "    intersect_areas = intersect_wh[..., 0] * intersect_wh[..., 1]\n",
    "    \n",
    "    true_areas = true_wh[..., 0] * true_wh[..., 1]\n",
    "    pred_areas = pred_wh[..., 0] * pred_wh[..., 1]\n",
    "\n",
    "    union_areas = pred_areas + true_areas - intersect_areas\n",
    "    iou_scores  = tf.truediv(intersect_areas, union_areas)    \n",
    "    return(iou_scores)\n",
    "\n",
    "def calc_IOU_pred_true_assigned(true_box_conf,\n",
    "                                true_box_xy, true_box_wh,\n",
    "                                pred_box_xy,  pred_box_wh):\n",
    "    ''' \n",
    "    == input ==\n",
    "    \n",
    "    true_box_conf : tensor of shape (N batch, N grid h, N grid w, N anchor )\n",
    "    true_box_xy   : tensor of shape (N batch, N grid h, N grid w, N anchor , 2)\n",
    "    true_box_wh   : tensor of shape (N batch, N grid h, N grid w, N anchor , 2)\n",
    "    pred_box_xy   : tensor of shape (N batch, N grid h, N grid w, N anchor , 2)\n",
    "    pred_box_wh   : tensor of shape (N batch, N grid h, N grid w, N anchor , 2)\n",
    "        \n",
    "    == output ==\n",
    "    \n",
    "    true_box_conf : tensor of shape (N batch, N grid h, N grid w, N anchor)\n",
    "    \n",
    "    true_box_conf value depends on the predicted values \n",
    "    true_box_conf = IOU_{true,pred} if objecte exist in this anchor else 0\n",
    "    '''\n",
    "    iou_scores        =  get_intersect_area(true_box_xy,true_box_wh,\n",
    "                                            pred_box_xy,pred_box_wh)\n",
    "    true_box_conf_IOU = iou_scores * true_box_conf\n",
    "    return(true_box_conf_IOU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>calc_IOU_pred_true_assigned</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "input\n",
      "******************************\n",
      "true_box_conf_tf = Tensor(\"strided_slice_6:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "true_box_xy_tf   = Tensor(\"strided_slice_4:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "true_box_wh_tf   = Tensor(\"strided_slice_5:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "pred_box_xy_tf   = Tensor(\"add:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "pred_box_wh_tf   = Tensor(\"mul:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "******************************\n",
      "ouput\n",
      "******************************\n",
      "true_box_conf_IOU.shape = (500, 13, 13, 4)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEmCAYAAABs7FscAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xm4XEWd//H3B8IeNIm5MCyBAIZB\nYDToFVlcUFwQlcVRJCoQRQOO4OAyGsEZFsUBFVB/IBgHDChEUEAQUEEgIEjQBEJIZAsQJBKTCAKJ\nIJrk+/ujqs1Jp29339vdt5PD5/U8/dxz6mx16nZ/u7pOnTqKCMzMrLzW6XYGzMyssxzozcxKzoHe\nzKzkHOjNzErOgd7MrOQc6M3MSs6B3kpD0hxJ+3Q7H2ZrGgd6W2tImifprVVp4yXdBhARu0TE1Ab7\nGC0pJA3pYFbN1igO9GZt5C8QWxM50FtpFGv8knaXNF3Ss5IWSjozr3Zr/vu0pKWS9pS0jqQvSXpM\n0iJJF0l6aWG/h+dlT0r676rjnCTpJ5J+KOlZYHw+9h2Snpa0QNLZktYv7C8k/YekhyQtkfRlSTvk\nbZ6VdFlxfbNWOdBbWX0L+FZEvATYAbgsp78x/x0WEUMj4g5gfH69GdgeGAqcDSBpZ+A7wIeALYCX\nAltVHetA4CfAMOBiYDnwaWAksCewL/AfVdvsB7wG2AP4PDApH2MUsCswroVzN1uFA72tbX6aa8pP\nS3qaFIRr+QfwckkjI2JpREyrs88PAWdGxCMRsRT4InBoboZ5H/CziLgtIv4O/A9QPUDUHRHx04hY\nERHPR8SMiJgWEcsiYh7wXeBNVducHhHPRsQcYDZwfT7+M8DPgd2aLxKz+hzobW1zUEQMq7xYvaZc\ncSSwI3C/pN9JenedfW4JPFaYfwwYAmyelz1eWRARzwFPVm3/eHFG0o6SrpH0p9yc81VS7b5oYWH6\n+RrzQ+vk16xfHOitlCLioYgYB2wGnA78RNImrF4bB3gC2LYwvw2wjBR8FwBbVxZI2gh4WfXhqubP\nBe4HxuSmo+MBDfxszFrjQG+lJOnDknoiYgXwdE5eDiwGVpDa4iumAJ+WtJ2koaQa+KURsYzU9v4e\nSXvlC6Qn0zhobwo8CyyVtBPwibadmNkAONBbWe0HzJG0lHRh9tCI+FtuejkVuD238+8BXAD8gNQj\n51Hgb8CxALkN/VjgR6Ta/RJgEfBCnWN/DvhgXvd7wKXtPz2z5skPHjFrXq7xP01qlnm02/kxa4Zr\n9GYNSHqPpI1zG/83gHuBed3NlVnzHOjNGjuQdMH2CWAMqRnIP4VtreGmGzOzknON3sys5BzozcxK\n7kUf6CXdLqnft5tL2lzSfZI26ES+bFWSJkv6Srfz0SpJn8iDrC2VVH3jlfWTpJ0lTe92PvqjG7Gj\nNIE+jyi4MPeMqKR9TNLUOtu8B1gSEXfn+X0lPZpHHPxAYb1hku6StGklLSIWAjcDEzpxPmui4tjv\n1n+S1gPOBN6eB1SrHkqhuG5IenlhfmdJV0t6Jo94ebOkvQrL95E0v8Z+pkr6WJvP4xBJv5H0XK3P\nl6RJkh6QtELS+BrLP52Hh3hG0gXFgJdH8rxX0jJJJzWRnS+TekL1ldfRuayek3S/qp5nULXu1yQ9\nnkcQfUzSCYVlI3Ol8Ml8/8UdkvYuLF+jY0dpAn02BPjPfqx/NOlGmYpvAu8h3WxzrqR1c/r/AqdF\nxJKq7S8GjhpgXrtOHRg7vVBmtrrNgQ2BOf3ZSNIOwO2kbp3bkcbfuRK4XtKe7c5kE54ifVZO62P5\nPaQxiO6qXiDpHcBE0oieo0l3KJ9cWGUuaTTPaxtlQtIWpBFHf1pntSnA3aRhK04gDYXR08e65wM7\n5WEr9gI+KOm9edlS4KNADzCcNKzGzwqfoTU7dkREKV6kfs0TSW/CYTntY8DUPtZfnzR41NaFtEcK\n038ijZOyO/CLPvYxBHgO2LaP5ZOBc0hv2iXAncAOheV7Ab8Dnsl/9yosm0qqrdyet70eGJmXnU16\n41Vey4CT8rItgctJt/o/CnyqsM+TSLf0/5B0i/7HgA1Ib9JK98FvAhvUOJdXkO4YXZ6P+XThHM8F\nrgP+Crw15/1jhW3HA7cV5ncCbsj/qweAQ5r4/04GzsvbLQFuKZZ7X2UJjADmA+/J80NJweTwBsfb\nCDiDNMDZM8BtwEZ52QGkYP10PtdXVL0PPwfMyttdSgruO+byiVx+NzU4fgAvz9M/AK6rsc65wK15\neh9gfo11VvlftPkz1+fnKy+/DRhflXYJ8NXC/L7An2ps+8PKe7rO/g8HflVn+Y6kO5g3LaT9Gji6\niXPbivTF+vkay9YhBfUANstpbY0d7X6VrUY/nfTG/lwT644BVkRE8efuIkmvkvQq0ngofyEFvk/V\n2kGksVDmAq+qc5xxpBrL8LzuqQCSRpC+AL5Nqm2cCVxb1W77QeAjpDfN+pXziohjIv30Hwq8Pufz\nKknrAD8j1ai2In2Ijsu1qIrqsdNPII2JPjafx+7Al2qc632kX0B35GMPq8rnqaQxXuo27eSmtRtI\nH/jNcvl8R9Iu9bbLPkT68hsJzMz5r1uWEfEUqSb2PUmbAWcBMyPiogbH+gZpvPi9SF8WnwdWSNqR\nVEs8jlS7u45Usys+KOQQUs1uO+CVpGD3IFA5x2ER8ZYmzrfibcCPa6RfBuwtaeN+7OufJE1UYcjn\n6tdA9tmEXUjvz4p7gM0HeL3i30gVhXrHeiRWrU3fw8r/w2pymSwlVQ42Ib1Pi8tnkSo8VwP/FxGL\n8qJOxI62KVughzRe+LF1fp5VDCPVDIuOJo2LMgk4jDQY1Y3AhpJ+mdv6qscVX5L31ZcrIuK3+R97\nMSmgArwLeCgifhBp3PIppBEP31PY9vsR8WBEPE/6UI8t7jif40+BYyNdZ3gt0BMRp0TE3yPiEdJY\nK4cWNltl7HRS8DwlIhZFxGLSl9Jhdc6nlqsi4va8z781WPfdwLyI+H4+77tIv0De18Rxro2IWyPi\nBdIX1J6SRtGgLCPielKgvDGvW/cnc/7C/CjwnxHxx4hYHhG/ycf9QM7HDRHxD9IXwkakL4SKb0fE\nE/lL5mdU/d8GYCRpnJ1qC0if4eED2WlEnBaFIZ+rX61kuI6hpF86FZXpTWus20itz3C9Y1WO1+ex\nIuK0vPzVpF9Sz1QtfyXwElLlplip6UTsaJvSPd8yImZLuobUjHNfnVX/QtU/PCJmkn4CV9r/ziA9\nIegWUg3uCeBWSdtG/v2V91Gv9vOnwvRzrBxnvHoMdPJ88elFfW1bubD3E+CSiPhRTt4W2LKqNrYu\n6edqxSpjp9fIx2M5rT+q91nPtsDrqvI4hFWvlTQ8TkQslfQUKa/NlOUk4BhSs0GfF0GzkaTmlodr\nLFvlWBGxQtLj1P+/9bc8q/2Z9HSraluwsva4DFivxjrrkR7CsqZYSgqUFZXpegG7L6t9hhscq3K8\nusfKn+278y/hk4HPVC3/GzBFqefMzIi4p0Oxo23KWKMHOBH4OKs/8q3oIUCS+lrnLOBLudb7b8D0\nSE8LWo/0k71yMfPlrPpTtFnVY6BDGgf9j01u//9Ib9hiM8vjwKNVNbNNI2L/wjrVt0LXGov9iT6O\n2ddt1NXpfwWKzQn/UpXHW6ryODQimhnKd1RlQmlwsRGsvLbQZ1nmC2PfBS4CPlHszdKHP5N+nu9Q\nY9kqx5KknK9m/28D8Svg/TXSDyH9QnsO+AMwMpdLMW/bsvqXYGX58UrdPGu+OnAekK5tFJsrXgUs\nbOLLt5ZZpHb4esfavtjjJR+v2YvhQ6j9HqhYj1WHu67odOzot1IG+oiYS7oIVrN9LK/zD9IHqPrn\nFJLeBmwYEdfkpEeBt+R25A1Y+YSh3UnNEDU/SA1cB+wo6YOShuQuWTsD1zTYDklH5Xx/MNJ46xW/\nBZ6V9AVJG0laV9Kukl5bZ3dTgC9J6pE0ktT09cM+1l0IbK3GD66eCbxXaSCwl5Oe9lRxDem8D5O0\nXn69VtIrGuwTYH9Jr8/H/zJwZ0Q8TuOyPD7//SipqeWier2DcpleAJwpactcjnsqdQO8DHhX7k63\nHvBZ0gW/3zSR/4E6GdhL0qmSRkjaVNKxpIuRX8h5/gPpYv/pkobmvP4XqaZf8zGKEfHVyrWeWq++\nMpPLY0NSIFxH0oa5LCrL18/LBayXl1dizUXAkUrdRYeTKiqTC9uul7ddBxiSt+3rf3UD8Oq8fq3z\ne5D0Xjwx7+dg0jWTy2uc0zqSjpI0XMnuwCdJzS9I2qPy3sufrS+QelHdWbWfwYgd/dffq7dr6ovU\n2+GthflRpFrZ1DrbvAv4eVXaBqQ3x7aFtH3z/heQBrSqpJ9DoVdLjf1PBr5SmN+HQs8I0oXUGaR2\nwBnA6wvLptJHz5W87AVW7XlzfF62JSl4/4n003ZapVxIvW5+WJXHDUkXMRfk17dJb9Ra57M+6aLn\nU8Cfa51jThtJ6iW0hNRr6CRW7XXzr3k/i0lv/JuAsQ3+v5NZ2etmKWns+O0alSXpgupfWNmDZd2c\npxMaHG8j0sW0P+Z93srKXjcHA7/P6bcAu9R5H/6zzEndCQMY0sT7+Z+9bvL8rqQvrmfz+U8tvl8K\n7/kf5//9n4FfAjt34LM2Puev+Jpc9d6tXr5PYflnSJWGZ4HvU+jllf/P1duOr5OXHwMfqLN8dM7P\n86QLt8X/zYeAOXl6HeAXpPf2UuBB8pPB8vI3kWrfS/I6twBv7FTsaPfrRT+omdINQJWLmf3ZbjPS\nP3u3aHwB0sw6QNLOwIXA7rGWBLNuxI6GgT73ariI1M66ApgUEd+S9HVSr4a/ky5afSQinpY0mnQR\ntNLtaVpEHN2Z7JuZWSPNBPotgC0ionIb7wzgINIDk2+KiGWSTgeIiC/kQH9NROza0Zxb6Uiaw+oX\nVQGOioiL1/bjVR37DcDPay2LOu3jZgPRsHtlRFTabomIJZLuA7aK1De5YhrN9YM261NENHPT1Fp7\nvKpj/5pCd1mzTupXP/pcW9+NqivNpN4MxQcgbyfpbtLFli/lN3X1viaQB/XZZJNNXrPTTjv1Jytm\nZi96M2bM+HNENLo5tPmLsbl/7i3AqRFxRSH9BKAXeG9ERO7WNTQinpT0GtKdm7tExLN97bu3tzem\nT1+rRho1M+s6STMiorfRek31o899ZC8HLq4K8keQbmn/UOWKd0S8EPnmh4iYQbpQW++mBjMz66CG\ngT7fXXc+cF9EnFlI3490s8YBke7Mq6T3VG5wkLQ9afCwR9qdcTMza04zbfR7kwbpuVfSzJx2POnG\nmg2AG9J3wT+7Ub4ROEXSMtKQtkdHGtzJzMy6oJleN7eRbmWudl0f619OjVuMzcysO0o51o2Zma3k\nQG9mVnIO9GZmJedAb2ZWcg70ZmYlV7pHCZqVxeiJ13bt2PNOe1fXjm3t5xq9mVnJOdCbmZWcm27M\nbDXdajZyk1FnuEZvZlZyDvRmZiXnQG9mVnIO9GZmJedAb2ZWcg70ZmYl18wTpkZJulnSfZLmSPrP\nnD5C0g2SHsp/h+d0Sfq2pLmSZkl6dadPwszM+tZMjX4Z8NmIeAWwB/BJSTsDE4EbI2IMcGOeB3gn\n6fGBY4AJwLltz7WZmTWtYaCPiAURcVeeXgLcB2wFHAhcmFe7EDgoTx8IXBTJNGCYpC3annMzM2tK\nv9roJY0GdgPuBDaPiAWQvgyAzfJqWwGPFzabn9Oq9zVB0nRJ0xcvXtz/nJuZWVOaDvSShpKeBXtc\nRDxbb9UaabFaQsSkiOiNiN6enp5ms2FmZv3UVKCXtB4pyF8cEVfk5IWVJpn8d1FOnw+MKmy+NfBE\ne7JrZmb91UyvGwHnA/dFxJmFRVcDR+TpI4CrCumH5943ewDPVJp4zMxs8DUzeuXewGHAvZJm5rTj\ngdOAyyQdCfwBeH9edh2wPzAXeA74SFtzbGZm/dIw0EfEbdRudwfYt8b6AXyyxXyZmVmb+M5YM7OS\nc6A3Mys5B3ozs5JzoDczKzkHejOzknOgNzMrOQd6M7OSc6A3Mys5B3ozs5JzoDczKzkHejOzknOg\nNzMrOQd6M7OSc6A3Mys5B3ozs5Jr5glTF0haJGl2Ie1SSTPza17lgSSSRkt6vrDsvE5m3szMGmvm\nCVOTgbOBiyoJEfGByrSkM4BnCus/HBFj25VBMzNrTTNPmLpV0uhay/LzZA8B3tLebJmZWbu02kb/\nBmBhRDxUSNtO0t2SbpH0hhb3b2ZmLWqm6aaeccCUwvwCYJuIeFLSa4CfStolIp6t3lDSBGACwDbb\nbNNiNszMrC8DrtFLGgK8F7i0khYRL0TEk3l6BvAwsGOt7SNiUkT0RkRvT0/PQLNhZmYNtNJ081bg\n/oiYX0mQ1CNp3Ty9PTAGeKS1LJqZWSua6V45BbgD+FdJ8yUdmRcdyqrNNgBvBGZJugf4CXB0RDzV\nzgybmVn/NNPrZlwf6eNrpF0OXN56tszWHKMnXtvtLJi1xHfGmpmVnAO9mVnJOdCbmZWcA72ZWck5\n0JuZlZwDvZlZyTnQm5mVnAO9mVnJOdCbmZWcA72ZWck50JuZlZwDvZlZyTnQm5mVnAO9mVnJOdCb\nmZVcMw8euUDSIkmzC2knSfqjpJn5tX9h2RclzZX0gKR3dCrjZmbWnGYeDj4ZOBu4qCr9rIj4RjFB\n0s6kJ0/tAmwJ/ErSjhGxvA15tRcxP/zDbOAa1ugj4lag2ccBHgj8KD8k/FFgLrB7C/kzM7MWtdJG\nf4ykWblpZ3hO2wp4vLDO/JxmZmZdMtBAfy6wAzAWWACckdNVY92otQNJEyRNlzR98eLFA8yGmZk1\nMqBAHxELI2J5RKwAvsfK5pn5wKjCqlsDT/Sxj0kR0RsRvT09PQPJhpmZNWFAgV7SFoXZg4FKj5yr\ngUMlbSBpO2AM8NvWsmhmZq1o2OtG0hRgH2CkpPnAicA+ksaSmmXmAUcBRMQcSZcBvweWAZ90jxsz\ns+5qGOgjYlyN5PPrrH8qcGormTIzs/bxnbFmZiXnQG9mVnIO9GZmJedAb2ZWcg70ZmYl50BvZlZy\nDvRmZiXnQG9mVnIO9GZmJedAb2ZWcg70ZmYl50BvZlZyDvRmZiXnQG9mVnIO9GZmJedAb2ZWcg0D\nvaQLJC2SNLuQ9nVJ90uaJelKScNy+mhJz0uamV/ndTLzZmbWWDM1+snAflVpNwC7RsQrgQeBLxaW\nPRwRY/Pr6PZk08zMBqphoI+IW4GnqtKuj4hleXYasHUH8mZmZm3Qjjb6jwI/L8xvJ+luSbdIekNf\nG0maIGm6pOmLFy9uQzbMzKyWlgK9pBOAZcDFOWkBsE1E7AZ8BrhE0ktqbRsRkyKiNyJ6e3p6WsmG\nmZnVMeBAL+kI4N3AhyIiACLihYh4Mk/PAB4GdmxHRs3MbGAGFOgl7Qd8ATggIp4rpPdIWjdPbw+M\nAR5pR0bNzGxghjRaQdIUYB9gpKT5wImkXjYbADdIApiWe9i8EThF0jJgOXB0RDxVc8dmZjYoGgb6\niBhXI/n8Pta9HLi81UyZmVn7+M5YM7OSc6A3Mys5B3ozs5JzoDczKzkHejOzknOgNzMrOQd6M7OS\nc6A3Mys5B3ozs5JzoDczKzkHejOzknOgNzMrOQd6M7OSazh6pVnR6InXdjsLZtZPrtGbmZVcU4Fe\n0gWSFkmaXUgbIekGSQ/lv8NzuiR9W9JcSbMkvbpTmTczs8aabbqZDJwNXFRImwjcGBGnSZqY578A\nvJP0CMExwOuAc/NfM7O6utU0OO+0d3XluIOlqRp9RNwKVD8S8EDgwjx9IXBQIf2iSKYBwyRt0Y7M\nmplZ/7XSRr95RCwAyH83y+lbAY8X1puf01YhaYKk6ZKmL168uIVsmJlZPZ24GKsaabFaQsSkiOiN\niN6enp4OZMPMzKC1QL+w0iST/y7K6fOBUYX1tgaeaOE4ZmbWglYC/dXAEXn6COCqQvrhuffNHsAz\nlSYeMzMbfE31upE0BdgHGClpPnAicBpwmaQjgT8A78+rXwfsD8wFngM+0uY8m5lZPzQV6CNiXB+L\n9q2xbgCfbCVTZmbWPr4z1sys5BzozcxKzoHezKzkHOjNzErOgd7MrOQc6M3MSs6B3sys5BzozcxK\nzoHezKzkHOjNzErOgd7MrOQc6M3MSs6B3sys5BzozcxKzoHezKzkmhqPvhZJ/wpcWkjaHvgfYBjw\ncaDyxO/jI+K6AefQzMxaMuBAHxEPAGMBJK0L/BG4kvREqbMi4httyaGZmbWkXU03+wIPR8Rjbdqf\nmZm1SbsC/aHAlML8MZJmSbpA0vBaG0iaIGm6pOmLFy+utYqZmbVBy4Fe0vrAAcCPc9K5wA6kZp0F\nwBm1touISRHRGxG9PT09rWbDzMz60I4a/TuBuyJiIUBELIyI5RGxAvgesHsbjmFmZgPUjkA/jkKz\njaQtCssOBma34RhmZjZAA+51AyBpY+BtwFGF5K9JGgsEMK9qmZmZDbKWAn1EPAe8rCrtsJZyZGZm\nbeU7Y83MSs6B3sys5BzozcxKzoHezKzkHOjNzErOgd7MrOQc6M3MSs6B3sys5BzozcxKzoHezKzk\nHOjNzErOgd7MrOQc6M3MSq6l0SutO0ZPvLbbWTCztYhr9GZmJddyjV7SPGAJsBxYFhG9kkYAlwKj\nSQ8fOSQi/tLqsczMrP/aVaN/c0SMjYjePD8RuDEixgA35nkzM+uCTjXdHAhcmKcvBA7q0HHMzKyB\ndlyMDeB6SQF8NyImAZtHxAKAiFggabM2HMfMrCO62cFh3mnv6vgx2hHo946IJ3Iwv0HS/c1sJGkC\nMAFgm222aUM2zMyslpabbiLiifx3EXAlsDuwUNIWAPnvohrbTYqI3ojo7enpaTUbZmbWh5YCvaRN\nJG1amQbeDswGrgaOyKsdAVzVynHMzGzgWm262Ry4UlJlX5dExC8k/Q64TNKRwB+A97d4HDMzG6CW\nAn1EPAK8qkb6k8C+rezbzMzaw3fGmpmVnAO9mVnJOdCbmZWcA72ZWck50JuZlZwDvZlZyTnQm5mV\nnAO9mVnJOdCbmZWcA72ZWck50JuZlZwDvZlZyTnQm5mVnAO9mVnJOdCbmZXcgMejlzQKuAj4F2AF\nMCkiviXpJODjwOK86vERcV2rGV0TdfOBwmZmzWrlwSPLgM9GxF35cYIzJN2Ql50VEd9oPXtmZtaq\nAQf6iFgALMjTSyTdB2zVroyZmVl7tKWNXtJoYDfgzpx0jKRZki6QNLwdxzAzs4FpOdBLGgpcDhwX\nEc8C5wI7AGNJNf4z+thugqTpkqYvXry41ipmZtYGLQV6SeuRgvzFEXEFQEQsjIjlEbEC+B6we61t\nI2JSRPRGRG9PT08r2TAzszoGHOglCTgfuC8iziykb1FY7WBg9sCzZ2ZmrWql183ewGHAvZJm5rTj\ngXGSxgIBzAOOaimHZmbWklZ63dwGqMaiUvaZNzNbW/nOWDOzknOgNzMrOQd6M7OSc6A3Mys5B3oz\ns5JzoDczKzkHejOzknOgNzMrOQd6M7OSa2UIhDWGn/RkZtY31+jNzErOgd7MrOQc6M3MSs6B3sys\n5BzozcxKzoHezKzkOhboJe0n6QFJcyVN7NRxzMysvo4EeknrAucA7wR2Jj1ecOdOHMvMzOrrVI1+\nd2BuRDwSEX8HfgQc2KFjmZlZHZ26M3Yr4PHC/HzgdcUVJE0AJuTZpZIeKCweCfy5Q3lbW7lMVuXy\nWJ3LZFVrRXno9JY237aZlToV6Gs9NDxWmYmYBEyqubE0PSJ6O5GxtZXLZFUuj9W5TFbl8lipU003\n84FRhfmtgSc6dCwzM6ujU4H+d8AYSdtJWh84FLi6Q8cyM7M6OtJ0ExHLJB0D/BJYF7ggIub0Yxc1\nm3Re5Fwmq3J5rM5lsiqXR6aIaLyWmZmttXxnrJlZyTnQm5mVXFcDfaNhEiRtIOnSvPxOSaMHP5eD\np4ny+Iyk30uaJelGSU31oV2bNTuUhqT3SQpJpe5O10x5SDokv0/mSLpksPM42Jr43Gwj6WZJd+fP\nzv7dyGdXRURXXqSLtA8D2wPrA/cAO1et8x/AeXn6UODSbuV3DSmPNwMb5+lPlLk8mi2TvN6mwK3A\nNKC32/nu8ntkDHA3MDzPb9btfK8BZTIJ+ESe3hmY1+18D/armzX6ZoZJOBC4ME//BNhXUq2bscqg\nYXlExM0R8VyenUa6P6HMmh1K48vA14C/DWbmuqCZ8vg4cE5E/AUgIhYNch4HWzNlEsBL8vRLeRHe\n09PNQF9rmISt+lonIpYBzwAvG5TcDb5myqPoSODnHc1R9zUsE0m7AaMi4prBzFiXNPMe2RHYUdLt\nkqZJ2m/QctcdzZTJScCHJc0HrgOOHZysrTk6NQRCMxoOk9DkOmXR9LlK+jDQC7ypoznqvrplImkd\n4Cxg/GBlqMuaeY8MITXf7EP6xfdrSbtGxNMdzlu3NFMm44DJEXGGpD2BH+QyWdH57K0Zulmjb2aY\nhH+uI2kI6WfXU4OSu8HX1LARkt4KnAAcEBEvDFLeuqVRmWwK7ApMlTQP2AO4usQXZJv9zFwVEf+I\niEeBB0iBv6yaKZMjgcsAIuIOYEPSgGcvGt0M9M0Mk3A1cESefh9wU+QrKiXUsDxyM8V3SUG+7G2v\n0KBMIuKZiBgZEaMjYjTpusUBETG9O9ntuGY+Mz8lXbRH0khSU84jg5rLwdVMmfwB2BdA0itIgX7x\noOayy7oW6HObe2WYhPuAyyJijqRTJB2QVzsfeJmkucBngNI+qarJ8vg6MBT4saSZkko9flCTZfKi\n0WR5/BJ4UtLvgZuB/4qIJ7uT485rskw+C3xc0j3AFGB8iSuMNXkIBDOzkvOdsWZmJedAb2ZWcg70\nZmYl50BvZlZyDvRmZiXnQG9mVnIO9LbWk3ScpI07fIzxks7u5DHaQdKUPBTvp/tYPlnS+/L0+pK+\nKelhSQ9JukrS1nnZaEmzq7Y9SdLnOn8W1m4O9LZGUtLs+/M4oGagl7Ru+3K1ZpP0L8BeEfHKiDir\niU2+ShpGYseIGEO6q/aKEo8Q+6LlQG+rybW5+yR9Lz+84npJG+VlY/OoiLMkXSlpeE6fKul0Sb+V\n9KCkN+T0/8t38c6UtFjSiTnmtnyCAAAEBUlEQVT9vyT9Lu/n5Krjfge4CxglaZykeyXNlnR6jbx+\nCtgSuFnSzTltab4z8k5gT0nz8nAASOqVNDVPbyLpgpyPuyXVGgK5aJSkXyg95OLEQh4+k/M3W9Jx\nOe21+dw2zMeZI2nXOmX++Xye90g6bSBlDVwPbJbL+g19HSvvY2PgI8CnI2I5QER8H3gBeEuDcrC1\nTbcHxPdrzXsBo4FlwNg8fxnw4Tw9C3hTnj4F+Gaengqckaf3B35Vtc9tgfvz37eTHgYhUmXjGuCN\n+bgrgD3yNluSxinpIY3KeBNwUI38zgNGFuYDOKTWctKon1Pz9FcL5zUMeBDYpI8yGQ8sIA2TvREw\nO+/rNcC9wCak4SnmALvlbb4CfAM4B/hinfJ+J/AbVj5UZsRAyjqX3+wG/9vJpHGjXgncXWP5WcCn\nau2LNNzv57r9/vSr/y/X6K0vj0bEzDw9Axgt6aXAsIi4JadfSArQFVcU168kStoQ+DFwTEQ8Rgr0\nbyc9CekuYCdWjrD4WERMy9OvJQXlxZHGNLm46nh9WQ5c3sR6bwcmSppJCp4bAtvUWf+GiHgyIp4n\nnevr8+vKiPhrRCzN6ZXa9CnA20hfCF+rs9+3At+P/FCZiHhqoGXdD6L2MNiV9L7GRvGYKWuhbo5H\nb2u24hDIy0m12Ga3Wc6q763zgCsi4ld5XsD/RsR3ixsrPRP4r8WkfuS36G+RmyOyZaxsptywav//\nHhEPNLnf6iAX1M/jCFItf7183L/2sV5fQbeevsq6WXOBbSVtGhFLCumvBn4GPAkMr9pmBPDoAI5l\nXeYavTUtIp4B/lJo/z0MuKXOJkj6JLBpRJxWSP4l8FFJQ/M6W0narMbmdwJvkjQyX1Qd18fxlpAu\nKvZlHqmJBeDfq/JxbOXio9Iw0PW8TdKIfL3iIOB20rNqD5K0saRNgIOBX+f1JwH/Tfolstr1hYLr\nSeWxcc7HiIGUdX9ExF9JvxLOrFywlnQ46aL2TfnXyQJJleF9RwD7Abe1Kw82eFyjt/46AjgvB6VH\nSBf06vkc8I/cPALpYe/nKY0LfkeOsUuBD5Nqp/8UEQskfZE03K6A6yLiqhrHmAT8XNKCiHhzjeUn\nA+dLOp705VHxZeCbwKwc7OcB765zLrcBPwBeDlwSedx7SZOB3+Z1/i8i7s5Bc1lEXJID6W8kvSUi\nbqreaUT8QtJYYLqkv5Med3c8/S/r/voi6RrCg5JWkK6hHBwRlV8XhwPnSDojz58cEQ+3OQ82CDxM\nsZlZybnpxsys5Nx0Y1Yg6R2s3p7+aEQc3IZ9/xup6afohYh4Xav7rnGsc4C9q5K/FamvvL3IuOnG\nzKzk3HRjZlZyDvRmZiXnQG9mVnIO9GZmJff/AXmY9Iciml/zAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1211c7a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "true_box_conf_IOU_tf = calc_IOU_pred_true_assigned(\n",
    "                            true_box_conf_tf,\n",
    "                            true_box_xy_tf, true_box_wh_tf,\n",
    "                            pred_box_xy_tf,  pred_box_wh_tf)\n",
    "with tf.Session() as sess:\n",
    "    true_box_conf_IOU = sess.run(true_box_conf_IOU_tf)\n",
    "print(\"*\"*30 + \"\\ninput\\n\" + \"*\"*30)    \n",
    "print(\"true_box_conf_tf = {}\".format(true_box_conf_tf))\n",
    "print(\"true_box_xy_tf   = {}\".format(true_box_xy_tf))\n",
    "print(\"true_box_wh_tf   = {}\".format(true_box_wh_tf))\n",
    "print(\"pred_box_xy_tf   = {}\".format(pred_box_xy_tf))\n",
    "print(\"pred_box_wh_tf   = {}\".format(pred_box_wh_tf))\n",
    "print(\"*\"*30 + \"\\nouput\\n\" + \"*\"*30) \n",
    "print(\"true_box_conf_IOU.shape = {}\".format(true_box_conf_IOU.shape))\n",
    "vec  = true_box_conf_IOU.flatten()\n",
    "pick = vec!=0\n",
    "vec  = vec[pick]\n",
    "plt.hist(vec)\n",
    "plt.title(\"Histogram\\nN (%) nonzero true_box_conf_IOU = {} ({:5.2f}%)\".format(np.sum(pick),\n",
    "                                                             100*np.mean(pick)))\n",
    "plt.xlabel(\"nonzero true_box_conf_IOU\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Calculate $\\text{max}_{i',j'}\n",
    " \\;\\;IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i',j'}}$\n",
    " \n",
    "For each predicted bounded box from (grid cell, anchor box), \n",
    "calculate the best IOU, regardless of the ground truth anchor box that each object gets assigned.\n",
    "\n",
    "This calculation uses true_boxes tensor. \n",
    "This tensor corresponds to the input <code>b_batch</code> from SimpleBatchGenerator, which is already introduced in [Part 2](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html).\n",
    "\n",
    "From the previous blog, I cite the <code>b_batch</code> description.\n",
    "\n",
    "<code>\n",
    "b_batch\n",
    "</code>\n",
    "\n",
    "\n",
    "The numpy array of shape <code>(BATCH_SIZE, 1, 1, 1, TRUE_BOX_BUFFER, 4)</code>.\n",
    "\n",
    "- <code>b_batch[iframe,1,1,1,ibuffer,:]</code> \n",
    "    contains <code>ibuffer</code>th object's <code>(center_x,center_y,center_w,center_h)</code> in <code>iframe</code>th frame. \n",
    "    \n",
    "- If <code>ibuffer</code> > N objects in <code>iframe</code>th frame, then the values are simply 0.\n",
    "\n",
    "- <code>TRUE_BOX_BUFFER</code> has to be some large number, so that the frame with the biggest number of objects can also record all objects. \n",
    "\n",
    "- The order of the objects do not matter. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_IOU_pred_true_best(pred_box_xy,pred_box_wh,true_boxes):   \n",
    "    '''\n",
    "    == input ==\n",
    "    pred_box_xy : tensor of shape (N batch, N grid h, N grid w, N anchor, 2)\n",
    "    pred_box_wh : tensor of shape (N batch, N grid h, N grid w, N anchor, 2)\n",
    "    true_boxes  : tensor of shape (N batch, N grid h, N grid w, N anchor, 2)\n",
    "    \n",
    "    == output == \n",
    "    \n",
    "    best_ious\n",
    "    \n",
    "    for each iframe,\n",
    "        best_ious[iframe,igridy,igridx,ianchor] contains\n",
    "        \n",
    "        the IOU of the object that is most likely included (or best fitted) \n",
    "        within the bounded box recorded in (grid_cell, anchor) pair\n",
    "        \n",
    "        NOTE: a same object may be contained in multiple (grid_cell, anchor) pair\n",
    "              from best_ious, you cannot tell how may actual objects are captured as the \"best\" object\n",
    "    '''\n",
    "    true_xy = true_boxes[..., 0:2]           # (N batch, 1, 1, 1, TRUE_BOX_BUFFER, 2)\n",
    "    true_wh = true_boxes[..., 2:4]           # (N batch, 1, 1, 1, TRUE_BOX_BUFFER, 2)\n",
    "    \n",
    "    pred_xy = tf.expand_dims(pred_box_xy, 4) # (N batch, N grid_h, N grid_w, N anchor, 1, 2)\n",
    "    pred_wh = tf.expand_dims(pred_box_wh, 4) # (N batch, N grid_h, N grid_w, N anchor, 1, 2)\n",
    "    \n",
    "    iou_scores  =  get_intersect_area(true_xy,\n",
    "                                      true_wh,\n",
    "                                      pred_xy,\n",
    "                                      pred_wh) # (N batch, N grid_h, N grid_w, N anchor, 50)   \n",
    "\n",
    "    best_ious = tf.reduce_max(iou_scores, axis=4) # (N batch, N grid_h, N grid_w, N anchor)\n",
    "    return(best_ious)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>calc_IOU_pred_true_best</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "input\n",
      "******************************\n",
      "true_box_wh_tf   = Tensor(\"strided_slice_5:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "pred_box_xy_tf   = Tensor(\"add:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "pred_box_wh_tf   = Tensor(\"mul:0\", shape=(500, 13, 13, 4, 2), dtype=float32)\n",
      "******************************\n",
      "ouput\n",
      "******************************\n",
      "best_ious.shape = (500, 13, 13, 4)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEmCAYAAACefMz8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3X+8VVWd//HXO8hf4Q8UdBIYryVW\n6KQmITWVFn4NdQynctJ+iGWRZpo1NdGPSdOccPphmb9ilAR1/DHmNxnF0FHJavx1TVMRTRJU8heG\nIPgjRT/zx1ont4dz7l3cw70HuO/n43Eed++1195r7X3POZ+91t5nbUUEZmZmJV7T7gqYmdm6w0HD\nzMyKOWiYmVkxBw0zMyvmoGFmZsUcNMzMrJiDhlkdSXMl7dXuepitjRw0rN+RtFDS3nVph0n6DUBE\n7BQRc7rZRoekkDSwF6tqttZx0DBbCzkY2drKQcOsTrUlImmMpE5JT0t6XNIPc7Yb8t+lklZIeoek\n10j6pqQHJT0haYakzSvbPTQv+7Okf60r53hJl0o6X9LTwGG57BslLZX0qKTTJG1Q2V5I+pyk+yUt\nl3SipDfmdZ6WdEk1v9ma4KBh1rUfAz+OiM2ANwKX5PT35L9bRMSgiLgROCy/3gu8ARgEnAYgaRRw\nBvAx4PXA5sCwurImAJcCWwAXAC8BXwSGAO8AxgGfq1tnPLA7MBb4F2BqLmMEsDNwSAv7brYKBw3r\nr36Rz+CXSlpK+kJv5EVgB0lDImJFRNzUxTY/BvwwIh6IiBXA14CDc1fTh4H/jojfRMQLwLeA+oHf\nboyIX0TEyxHxXETcFhE3RcTKiFgI/BTYs26dkyPi6YiYC9wNXJ3LXwZcBexWfkjMuuegYf3VgRGx\nRe3FqmfwNYcDOwL3SrpV0j90sc1tgQcr8w8CA4Ft8rKHawsi4lngz3XrP1ydkbSjpCskPZa7rP6N\n1Oqoerwy/VyD+UFd1NdstTlomHUhIu6PiEOArYGTgUslvY5VWwkAjwDbVeb/FlhJ+iJ/FBheWyBp\nY2Cr+uLq5s8E7gVG5u6xrwPq+d6Ytc5Bw6wLkj4uaWhEvAwszckvAYuBl0nXLmouBL4oaXtJg0gt\ng4sjYiXpWsUBkt6ZL05/m+4DwKbA08AKSW8GjlxjO2bWQw4aZl0bD8yVtIJ0UfzgiHg+dy+dBPw2\nXxcZC0wDziPdWbUAeB44GiBfczgauIjU6lgOPAH8pYuyvwx8NOf9D+DiNb97ZqtHfgiTWd/LLZGl\npK6nBe2uj1kptzTM+oikAyRtkq+JfB+4C1jY3lqZrR4HDbO+M4F0sfwRYCSpq8tNfVunuHvKzMyK\nuaVhZmbFHDTMzKyYg8ZaRtJvJa320A+StpE0T9KGvVGvtVF1OPM2lf9uSfe1q/z+SNJnJf2o3fVo\nhaQfSjqi3fXoKQeNXpJHMH083ylTS/u0pDldrHMAsDwibs/z4yQtyCOcfqSSbwtJv5O0aS0tIh4H\nrgcm9cb+9Cd59NgdussXEb+OiDf1RZ1aIWlDSefkEXaXS7pd0r6V5bVng6yovP61snxu3bKVkv67\nsjwkPVNZfnZl2fGSXqxb/w2V5btKuk3Ss/nvrl3sxwbAN4Hv5fl31213Ra7LhyrrvCEPxbJc0pOS\n/r3JtneUdLmkxZKWSJot6U11eb6Yh3RZJmla7QRN0kBJF+Xf61xV/VxK+oakL9YV9z3gG+vqCMQO\nGr1rIPCF1ch/BOnHYTU/Ag4g/cDsTEkDcvp3gSkRsbxu/QuAz/awrm0nP0OitwwkjWu1J2l03X8F\nLpHUUZevNmLvoIg4sZaYH0o1KCIGkX6l/hDwX3Xr7lJZ99N1yy6uLBsUEQ/AX4PA5cD5wGBgOnB5\nF1+mE4B7I+JPuV6/rm4X+AdgBfDLyvavAa4D/oY0jMv5Tba9BTATeBNprLBbct3I23o/MJk00nAH\naSSAb+fFHyQNATOE9Av+z+Z1tid9fn9SLSgiHiUND/OBJnVZu0WEX73wIt1/PxlYQvowAnwamNMk\n/wakAeaGV9IeqEw/Rhr/aAzwyybbGAg8C2zXZPm5wOnAlaRfGd8MvLGy/J3ArcCy/PedlWVzgBOB\n3+Z1rwaG5GWnkT6stddK4Pi8bFvg56RhNxYAx1S2eTxpeI3zSR+2TwMbkoJl7dbUHwEbNtmfw3J9\nfpLrfC8wrrJ8c+Ac0i+w/wR8BxiQl+0A/Cqv9yTpiw3Sr7kDeCbvy0e6+B/vBSyqzL8lH6elwFzg\nA3XH79N1df9NnhZwCukX4suAO4Gde/n9eSfwoTzdkfd5YMF6e+bj8rpKWgA7NMl/PHB+k2X75P+L\nKmkPAeOb5J8GfLOLuv0M+FllfhLw6x4eny3zfm2V5/8T+LfK8nHAY3n6q8Bn8/QRwBl5+r+BdzXZ\n/jeqdV2XXm5p9K5O0pfFlwvyjgRejohFlbQnJO0iaRfSOEdPkb5Ej2m0gUhjHM0HduminENIZ0iD\nc96TACRtSQomp5IG0vshcKWk6qB6HwU+SQpeG9T2KyI+H6+c7b0r1/NySa8hfXB+T3p2xDjg2HzW\nVlP/DIlvkJ4NsWvejzGkLolm9gAeIJ3lHQdclvcF0pnrSlKA2I30JVU7Cz6RFPgGk85Af5L3pfac\njNqZc9HQHZJem/f16nx8jgYuqO/iaGIf0vM5diQdh4+w6gi4tXLOUGVI97rXnYV13SaXNbdu0YOS\nFkn6maT60XRrJgKXRsQzdek35K6byxq0YA7IXT5zJVXHz9oJuDPyt2h2Z05v5O+AhteQJG1CGn5+\neiV5LLAwdxk9KWmOpL9rsu167yEFhdr/YSfS+7jm98A2+fNxN/C+3LJ5L2nYmX8EnoyIZtfc5tH1\n53St5aDR+74FHC1paDf5tiCdwVcdQRrvaCrwCdKAddcCG+U+1+sl1T9fYXneVjOXRcQtOcBcQPpy\nBtgfuD8izov0/IYLSWfuB1TW/VlE/CEiniM9jOhV/c95H38BHB3puszbgaERcUJEvBCpW+I/gIMr\nq73qGRKkZ1KcEBFPRMRiUoD7RBf78wTwo4h4MX/B3wfsn78Y9wWOjYhnIuIJ0tl8rewXSSPSbhtp\nLKlWL6iPJQ1DPiXv63XAFZQ9BOlFUrfPm0ln3fMidWGsIiI+F5Uh3eteb+2uoBzcLgCmR8S9OflJ\n0v9qO9IDnTbNeerXrX0xn1u3aE9Sa+XNpNbhFZWuxktILbChwGeAb0mqHZNBpJZV1bJcfiONPiM1\nH8r78atK2nDS//tUUov3Srru/gJA0nBSi/xLleT6utamNwVmkVrRnTn9ItIJzFclnSTphhzsq+V2\n9zldazlo9LKIuJv05TG5m6xPUfdhiYg7ImKviNgDuAf4FGnk1LNJX6afBM6TVB0tdVNeGY21kccq\n08/yyvMW6p8FQZ6vPl2u2bq1L6NLgf+MiIty8nbAtnr1w46+TuozrnnVMyQa1OPBnNbMn+rOVGv5\ntwNeCzxaKfunpFYApKfcCbglnwF/qosySmwLPBxpNNxqXeqfzreKHGBOI31RPS5pqqTNWqzPKnLL\n7zzgBeDzlfJXRERnPll4PC/bp0EdPkjqbq1+MRMRN+RAuZR0DW97UqAgIu6JiEci4qWI+F/SSdCH\n86orgPoyNqN5YFjlM1IxEZhR9154jtQFeFWkB199n9SKfkuTbdROfK4mdTFdWFlUX9fa9PJIJkfE\nWyNiEumzfhYwOr/2JLXMq++x7j6nay0Hjb5xHOksq6svkPsBSWqW5xRSf+5zpGZ6Z6Snub2WdBZX\nu5C8A69uRpeqfxYEpOdB/Klw/Z+QPuzVrqSHgQV1Z8ObRsR+lTz1QxI0eibFI12UO6wuaNbyP0wa\nQXZIpezNImIngIh4LCI+ExHbki5cnqGCO6a68AgwIn8xV+tSO37PAJtUlv1NdeWIODUidid1g+wI\nfKVRIZLOanDHUO1V391UXU+k6zvbkK5lvNjFvtT+J/VDtzf6Ym62frNh36vL5gJvrfv/vZVVu81q\n7iQdm1eRNIJ0fWlGg/zFQ15IGkwKGDMj4qS6xXN5dXfSLsDjle6r2jZ2Jl0bnEr6nN6Wj9etpH2r\neQs9+5y2nYNGH4iI+aRhrRtei8h5XgT+h1Uf54mk/wdsFBFX5KQFpD7UnUgXjmtv3DHAwoiobzGU\nmAXsKOmj+RbCjwCjSK2kLkn6bK73R+vOtG8Bnpb0VUkbSxogaWdJb+9icxcC35Q0NPerf4vmd7xA\najkcI+m1kg4ifRhn5e6dq4EfSNpM0mskvbHWnSfpoNwNAekMNkjPyYD00KQ3sHpuJgWGf8l12YvU\ntVdrdd0BfFBpwMIdSE8EJNfl7ZL2yK21Z0hDqr9EAxFxRLz6TqTqq9m1AEgPdHoLcEA+8firXPab\n8jHaitSdMyfSI2NreYaT+uun1627k9JtswOURu79ASlQzsvLJ0garGQM6TNQuytpTt7PY5RuC661\nfq5rsg+zaPD5IHVf/m9E/LEu/XxgrKS9le48PJbUhTWvfgO5VTUb+G1ENOoVmAEcLmlUDi7fpK6b\nLge/04Ev5M/BAuBduVtqT9K1t5o9SY/jXffUXxn3a828SHdP7V2ZH0H6MpjTxTr7A1fVpW1I+sLZ\nrpI2Lm//UdKgd7X006ncndRg++cC36nM78Wr7/55F3AbqV/2Nip3ftD13T9zSGf11Tuovp6XbUsK\nBI+Rvpxvqh0XGtxZA2xE+tJ6NL9OJQXMRvtzGOnuqdNynf8A7FNZvjnpy3JRXn577XgB/076clsB\n/BGYVFnviFz2UuCfujie9cdvJ165I+se4B8ry4aQgtjyXOfjK8dvHOmseAXpS+0CYNAafC9uRwqK\nz9f9jz6Wlx9C+oJ7Ju/3DOBv6rbxNRrciQS8j3Qd6RnS9aVfkIZ7ry2/kHRSs4J0jeyYuvV3y++1\n54DfAbt1sR+vJd1dtW1d+r3A4U3W+SDpho+n8/t0p8qyqyrv04m8+q652utvK/m/RDqheJp0p9aG\ndWV9Cji9Mj+QdNKwjBSQNs3pr8/vyQ3W1P+4L18esHAto/QL59qF5NVZb2vSF9ZuEfF8r1TOrM0k\nTQJGRcSx7a5LT0n6AfDHiDij3XXpCQcNMzMr5msaZl2Q9PUmF53Xzf5osxa5pWFmZsXWu7F+hgwZ\nEh0dHe2uhpnZOuW22257MiK6+xHy+hc0Ojo66OzsbHc1zMzWKZKKbtX3NQ0zMyvmoGFmZsUcNMzM\nrJiDhpmZFXPQMDOzYg4aZmZWzEHDzMyKOWiYmVkxBw0zMyu23v0ivBUdk69sS7kLp+zflnLNzFaX\nWxpmZlbMQcPMzIo5aJiZWTEHDTMzK+agYWZmxRw0zMysmIOGmZkVc9AwM7NiDhpmZlas26AhaZqk\nJyTdXUnbUtI1ku7PfwfndEk6VdJ8SXdKeltlnYk5//2SJlbSd5d0V17nVEnqqgwzM2ufkpbGucD4\nurTJwLURMRK4Ns8D7AuMzK9JwJmQAgBwHLAHMAY4rhIEzsx5a+uN76YMMzNrk26DRkTcACypS54A\nTM/T04EDK+kzIrkJ2ELS64H3A9dExJKIeAq4Bhifl20WETdGRAAz6rbVqAwzM2uTnl7T2CYiHgXI\nf7fO6cOAhyv5FuW0rtIXNUjvqoxVSJokqVNS5+LFi3u4S2Zm1p01fSFcDdKiB+mrJSKmRsToiBg9\ndOjQ1V3dzMwK9TRoPJ67lsh/n8jpi4ARlXzDgUe6SR/eIL2rMszMrE16GjRmArU7oCYCl1fSD813\nUY0FluWupdnAPpIG5wvg+wCz87Llksbmu6YOrdtWozLMzKxNun0Ik6QLgb2AIZIWke6CmgJcIulw\n4CHgoJx9FrAfMB94FvgkQEQskXQicGvOd0JE1C6uH0m6Q2tj4Kr8oosyzMysTboNGhFxSJNF4xrk\nDeCoJtuZBkxrkN4J7Nwg/c+NyjAzs/bxL8LNzKyYg4aZmRVz0DAzs2IOGmZmVsxBw8zMijlomJlZ\nMQcNMzMr5qBhZmbFHDTMzKyYg4aZmRVz0DAzs2IOGmZmVsxBw8zMijlomJlZMQcNMzMr5qBhZmbF\nHDTMzKyYg4aZmRVz0DAzs2IOGmZmVsxBw8zMijlomJlZMQcNMzMr5qBhZmbFHDTMzKyYg4aZmRVz\n0DAzs2IOGmZmVsxBw8zMijlomJlZsZaChqQvSpor6W5JF0raSNL2km6WdL+kiyVtkPNumOfn5+Ud\nle18LaffJ+n9lfTxOW2+pMmt1NXMzFrX46AhaRhwDDA6InYGBgAHAycDp0TESOAp4PC8yuHAUxGx\nA3BKzoekUXm9nYDxwBmSBkgaAJwO7AuMAg7Jec3MrE1a7Z4aCGwsaSCwCfAo8D7g0rx8OnBgnp6Q\n58nLx0lSTr8oIv4SEQuA+cCY/JofEQ9ExAvARTmvmZm1SY+DRkT8Cfg+8BApWCwDbgOWRsTKnG0R\nMCxPDwMezuuuzPm3qqbXrdMsfRWSJknqlNS5ePHinu6SmZl1o5XuqcGkM//tgW2B15G6kupFbZUm\ny1Y3fdXEiKkRMToiRg8dOrS7qpuZWQ+10j21N7AgIhZHxIvAZcA7gS1ydxXAcOCRPL0IGAGQl28O\nLKmm163TLN3MzNqklaDxEDBW0ib52sQ44B7geuDDOc9E4PI8PTPPk5dfFxGR0w/Od1dtD4wEbgFu\nBUbmu7E2IF0sn9lCfc3MrEUDu8/SWETcLOlS4HfASuB2YCpwJXCRpO/ktHPyKucA50maT2phHJy3\nM1fSJaSAsxI4KiJeApD0eWA26c6saRExt6f1NTOz1vU4aABExHHAcXXJD5DufKrP+zxwUJPtnASc\n1CB9FjCrlTqamdma41+Em5lZMQcNMzMr5qBhZmbFHDTMzKyYg4aZmRVz0DAzs2IOGmZmVsxBw8zM\nijlomJlZMQcNMzMr5qBhZmbFHDTMzKyYg4aZmRVz0DAzs2IOGmZmVsxBw8zMijlomJlZMQcNMzMr\n5qBhZmbFHDTMzKyYg4aZmRVz0DAzs2IOGmZmVsxBw8zMijlomJlZMQcNMzMr5qBhZmbFHDTMzKyY\ng4aZmRVrKWhI2kLSpZLulTRP0jskbSnpGkn357+Dc15JOlXSfEl3SnpbZTsTc/77JU2spO8u6a68\nzqmS1Ep9zcysNa22NH4M/DIi3gzsAswDJgPXRsRI4No8D7AvMDK/JgFnAkjaEjgO2AMYAxxXCzQ5\nz6TKeuNbrK+ZmbVgYE9XlLQZ8B7gMICIeAF4QdIEYK+cbTowB/gqMAGYEREB3JRbKa/Pea+JiCV5\nu9cA4yXNATaLiBtz+gzgQOCqntZ5bdUx+cq2lb1wyv5tK9vM1j2ttDTeACwGfibpdklnS3odsE1E\nPAqQ/26d8w8DHq6svyindZW+qEG6mZm1SStBYyDwNuDMiNgNeIZXuqIaaXQ9InqQvuqGpUmSOiV1\nLl68uOtam5lZj7USNBYBiyLi5jx/KSmIPJ67nch/n6jkH1FZfzjwSDfpwxukryIipkbE6IgYPXTo\n0BZ2yczMutLjoBERjwEPS3pTThoH3APMBGp3QE0ELs/TM4FD811UY4FluftqNrCPpMH5Avg+wOy8\nbLmksfmuqUMr2zIzszbo8YXw7GjgAkkbAA8AnyQFokskHQ48BByU884C9gPmA8/mvETEEkknArfm\nfCfULooDRwLnAhuTLoCvdxfBzczWJS0FjYi4AxjdYNG4BnkDOKrJdqYB0xqkdwI7t1JHMzNbc/yL\ncDMzK+agYWZmxRw0zMysmIOGmZkVc9AwM7NiDhpmZlbMQcPMzIo5aJiZWTEHDTMzK+agYWZmxRw0\nzMysmIOGmZkVc9AwM7NiDhpmZlbMQcPMzIo5aJiZWTEHDTMzK+agYWZmxRw0zMysmIOGmZkVc9Aw\nM7NiDhpmZlbMQcPMzIo5aJiZWTEHDTMzK+agYWZmxRw0zMysmIOGmZkVc9AwM7NiDhpmZlas5aAh\naYCk2yVdkee3l3SzpPslXSxpg5y+YZ6fn5d3VLbxtZx+n6T3V9LH57T5kia3WlczM2vNmmhpfAGY\nV5k/GTglIkYCTwGH5/TDgaciYgfglJwPSaOAg4GdgPHAGTkQDQBOB/YFRgGH5LxmZtYmLQUNScOB\n/YGz87yA9wGX5izTgQPz9IQ8T14+LuefAFwUEX+JiAXAfGBMfs2PiAci4gXgopzXzMzapNWWxo+A\nfwFezvNbAUsjYmWeXwQMy9PDgIcB8vJlOf9f0+vWaZa+CkmTJHVK6ly8eHGLu2RmZs0M7OmKkv4B\neCIibpO0Vy25QdboZlmz9EYBLRqkERFTgakAo0ePbpjHGuuYfGVbyl04Zf+2lGtmrelx0AD+HviA\npP2AjYDNSC2PLSQNzK2J4cAjOf8iYASwSNJAYHNgSSW9prpOs3QzM2uDHndPRcTXImJ4RHSQLmRf\nFxEfA64HPpyzTQQuz9Mz8zx5+XURETn94Hx31fbASOAW4FZgZL4ba4Ncxsye1tfMzFrXSkujma8C\nF0n6DnA7cE5OPwc4T9J8UgvjYICImCvpEuAeYCVwVES8BCDp88BsYAAwLSLm9kJ9zcys0BoJGhEx\nB5iTpx8g3flUn+d54KAm658EnNQgfRYwa03U0czMWudfhJuZWTEHDTMzK+agYWZmxRw0zMysmIOG\nmZkVc9AwM7NiDhpmZlbMQcPMzIo5aJiZWTEHDTMzK+agYWZmxRw0zMysmIOGmZkVc9AwM7NiDhpm\nZlbMQcPMzIo5aJiZWTEHDTMzK+agYWZmxRw0zMysmIOGmZkVc9AwM7NiDhpmZlbMQcPMzIoNbHcF\nrH/qmHxl28peOGX/tpVttq5zS8PMzIo5aJiZWTEHDTMzK+agYWZmxRw0zMysWI+DhqQRkq6XNE/S\nXElfyOlbSrpG0v357+CcLkmnSpov6U5Jb6tsa2LOf7+kiZX03SXdldc5VZJa2VkzM2tNKy2NlcA/\nR8RbgLHAUZJGAZOBayNiJHBtngfYFxiZX5OAMyEFGeA4YA9gDHBcLdDkPJMq641vob5mZtaiHgeN\niHg0In6Xp5cD84BhwARges42HTgwT08AZkRyE7CFpNcD7weuiYglEfEUcA0wPi/bLCJujIgAZlS2\nZWZmbbBGrmlI6gB2A24GtomIRyEFFmDrnG0Y8HBltUU5rav0RQ3SG5U/SVKnpM7Fixe3ujtmZtZE\ny0FD0iDg58CxEfF0V1kbpEUP0ldNjJgaEaMjYvTQoUO7q7KZmfVQS0FD0mtJAeOCiLgsJz+eu5bI\nf5/I6YuAEZXVhwOPdJM+vEG6mZm1SSt3Twk4B5gXET+sLJoJ1O6AmghcXkk/NN9FNRZYlruvZgP7\nSBqcL4DvA8zOy5ZLGpvLOrSyLTMza4NWBiz8e+ATwF2S7shpXwemAJdIOhx4CDgoL5sF7AfMB54F\nPgkQEUsknQjcmvOdEBFL8vSRwLnAxsBV+WVmZm3S46AREb+h8XUHgHEN8gdwVJNtTQOmNUjvBHbu\naR3NzGzN8i/CzcysmIOGmZkVc9AwM7NiDhpmZlbMQcPMzIo5aJiZWTEHDTMzK9bKj/vM1kkdk69s\nS7kLp+zflnLN1iS3NMzMrJiDhpmZFXPQMDOzYg4aZmZWzEHDzMyKOWiYmVkxBw0zMyvmoGFmZsUc\nNMzMrJiDhpmZFXPQMDOzYg4aZmZWzAMWmvWRdg2UCB4s0dYctzTMzKyYg4aZmRVz0DAzs2IOGmZm\nVsxBw8zMivnuKbN+wI+4tTXFLQ0zMyvmoGFmZsUcNMzMrNhaf01D0njgx8AA4OyImNLmKplZIf8K\nfv2zVrc0JA0ATgf2BUYBh0ga1d5amZn1X2t7S2MMMD8iHgCQdBEwAbinrbUys7We7xjrHWt70BgG\nPFyZXwTsUZ9J0iRgUp5dIem+HpY3BHiyh+uuT3wcEh+HxMchKToOOrkPatI7tivJtLYHDTVIi1US\nIqYCU1suTOqMiNGtbmdd5+OQ+DgkPg6Jj0OyVl/TILUsRlTmhwOPtKkuZmb93toeNG4FRkraXtIG\nwMHAzDbXycys31qru6ciYqWkzwOzSbfcTouIub1YZMtdXOsJH4fExyHxcUh8HABFrHKJwMzMrKG1\nvXvKzMzWIg4aZmZWrF8GDUnjJd0nab6kyQ2Wbyjp4rz8ZkkdfV/L3ldwHL4k6R5Jd0q6VlLRfdzr\nmu6OQyXfhyWFpPXytsuS4yDpn/J7Yq6k/+zrOvaFgs/F30q6XtLt+bOxXzvq2TYR0a9epAvqfwTe\nAGwA/B4YVZfnc8BZefpg4OJ217tNx+G9wCZ5+sj+ehxyvk2BG4CbgNHtrneb3g8jgduBwXl+63bX\nu03HYSpwZJ4eBSxsd7378tUfWxp/HZokIl4AakOTVE0ApufpS4Fxkhr90HBd1u1xiIjrI+LZPHsT\n6Xcy65uS9wPAicC/A8/3ZeX6UMlx+AxwekQ8BRART/RxHftCyXEIYLM8vTn97Ldj/TFoNBqaZFiz\nPBGxElgGbNUntes7Jceh6nDgql6tUXt0exwk7QaMiIgr+rJifazk/bAjsKOk30q6KY9Avb4pOQ7H\nAx+XtAiYBRzdN1VbO6zVv9PoJSVDkxQNX7KOK95HSR8HRgN79mqN2qPL4yDpNcApwGF9VaE2KXk/\nDCR1Ue1FanX+WtLOEbG0l+vWl0qOwyHAuRHxA0nvAM7Lx+Hl3q9e+/XHlkbJ0CR/zSNpIKkJuqRP\natd3ioZokbQ38A3gAxHxlz6qW1/q7jhsCuwMzJG0EBgLzFwPL4aXfi4uj4gXI2IBcB8piKxPSo7D\n4cAlABFxI7ARaTDDfqE/Bo2SoUlmAhPz9IeB6yJf9VqPdHsccrfMT0kBY33sv4ZujkNELIuIIRHR\nEREdpGs7H4iIzvZUt9eUfC5+Qbo5AklDSN1VD/RpLXtfyXF4CBgHIOktpKCxuE9r2Ub9LmjkaxS1\noUnmAZdExFxJJ0j6QM52DrCVpPnAl4Cmt2GuqwqPw/eAQcB/SbpD0no37lfhcVjvFR6H2cCfJd0D\nXA98JSL+3J4a947C4/DPwGck/R64EDhsPTypbMrDiJiZWbF+19IwM7Oec9AwM7NiDhpmZlbMQcPM\nzIo5aJiZWTEHDTMzK+agYdaXofmzAAADkElEQVQHJC3MP4hrZRsdkj7aTZ7Rkk5tpRyzrjhomPWQ\nkr78DHUAXQaNiOiMiGP6pjrWHzlo2Dopn3XPk/Qf+YFAV0vaOC/bNY/Ceqek/y9pcE6fI+lkSbdI\n+oOkd+f0s/Mv3u+QtFjScTn9K5Juzdv5dl25ZwC/A0ZIOkTSXZLulnRyF9X+Si77Fkk75O0NlfTz\nXM6tkv4+p+9ZqdPtkjYFpgDvzmlfbHJc9pJ0RZ7eUtIvcv1vkvTWnH68pC9X1rk779frJF0p6fc5\n7SMt/ItsPeWgYeuykaTnO+wELAU+lNNnAF+NiLcCdwHHVdYZGBFjgGNr6RHx6YjYlfTchD8D50ra\nJ29/DLArsLuk9+RtvAmYERG7AS8CJwPvy/neLunAJvV9Opd9GvCjnPZj4JSIeHuu/9k5/cvAUble\n7waeIw1n8+uI2DUiTik4Pt8Gbs/H4ev5uHRlPPBIROwSETsDvywow/oZBw1bly2IiDvy9G1Ah6TN\ngS0i4lc5fTrwnso6l1Xz1xIlbQT8F/D5iHgQ2Ce/bie1KN7MKyO6PhgRN+XptwNzImJxHrfogrry\nqi6s/H1Hnt4bOE3SHaSB8TbLrYrfAj+UdEzen5UlB6TOu4DzACLiOtJ4apt3kf8uYO/cGnt3RCzr\nQZm2nuuPz9Ow9Ud1qPaXgI1XY52XePX7/yzgsoj4nzwv4LsR8dPqykrPi3+mmrQa9Y0G068B3hER\nz9XlnSLpSmA/4KY8RP3qavZsiJW8+oRxI4CI+IOk3XOZ35V0dUSc0INybT3mloatV/LZ8VO16xXA\nJ4BfdbEKko4CNo2IKZXk2cCnJA3KeYZJ2rrB6jcDe0oaImkA6QE9zcr7SOXvjXn6atKoqrW67Jr/\nvjEi7oqIk4FOUktnOen5HqVuAD6Wt7cX8GREPA0sBN6W098GbJ+ntwWejYjzge/X8phVuaVh66OJ\nwFmSNiE97+GT3eT/MvBi7iICOCsizsrPSrhR6fHwK4CPk1oofxURj0r6GmmocAGzIuLyJuVsKOlm\n0snaITntGOB0SXeSPo83AEcAx0p6by7vHtKjdl8GVuYhuc8tuK5xPPCzvO1neeUZMT8HDs37eyvw\nh5z+d8D3JL1MulZzZDfbt37IQ6ObmVkxd0+ZmVkxd0+ZrYMkvZ90q2/Vgoj4x3bUx/oPd0+ZmVkx\nd0+ZmVkxBw0zMyvmoGFmZsUcNMzMrNj/AWeteBGZTtDyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11f860400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "true_boxes_tf = tf.constant(b_batch,dtype=\"float32\")\n",
    "best_ious_tf = calc_IOU_pred_true_best(pred_box_xy_tf,\n",
    "                                       pred_box_wh_tf,\n",
    "                                       true_boxes_tf)\n",
    "with tf.Session() as sess:\n",
    "    best_ious = sess.run(best_ious_tf)\n",
    "    \n",
    "print(\"*\"*30 + \"\\ninput\\n\" + \"*\"*30)    \n",
    "print(\"true_box_wh_tf   = {}\".format(true_box_wh_tf))\n",
    "print(\"pred_box_xy_tf   = {}\".format(pred_box_xy_tf))\n",
    "print(\"pred_box_wh_tf   = {}\".format(pred_box_wh_tf))\n",
    "print(\"*\"*30 + \"\\nouput\\n\" + \"*\"*30) \n",
    "print(\"best_ious.shape = {}\".format(best_ious.shape))\n",
    "vec  = best_ious.flatten()\n",
    "pick = vec!=0\n",
    "vec  = vec[pick]\n",
    "plt.hist(vec)\n",
    "plt.title(\"Histogram\\nN (%) nonzero best_ious = {} ({:5.2f}%)\".format(np.sum(pick),\n",
    "                                                             100*np.mean(pick)))\n",
    "plt.xlabel(\"nonzero best_ious\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7: Calculate $\\lambda_{\\text{noobj}}L_{i,j}^{\\text{noobj}}$ and $\\lambda_{\\text{obj}}L_{i,j}^{\\text{obj}}$\n",
    "For each grid cell, calculate no object mask. \n",
    "$$\n",
    "\\begin{array}{rl}\n",
    "L_{i,j}^{\\text{noobj}}\n",
    "& = \n",
    "\\begin{cases}\n",
    " 1 \\;\\;\\text{if}\\;\\;\\text{max}_{i',j'}\n",
    " \\;\\;IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i',j'}} < 0.6 \\;\\text{and}\\; C_{i,j} = 0\\\\\n",
    " 0\\;\\;\\text{else}\\\\\n",
    "\\end{cases}\n",
    "\\end{array}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_conf_mask(best_ious, true_box_conf, true_box_conf_IOU,LAMBDA_NO_OBJECT, LAMBDA_OBJECT):    \n",
    "    '''\n",
    "    == input == \n",
    "    \n",
    "    best_ious           : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    true_box_conf       : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    true_box_conf_IOU   : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    LAMBDA_NO_OBJECT    : 1.0\n",
    "    LAMBDA_OBJECT       : 5.0\n",
    "    \n",
    "    == output ==\n",
    "    conf_mask : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    \n",
    "    conf_mask[iframe, igridy, igridx, ianchor] = 0\n",
    "               when there is no object assigned in (grid cell, anchor) pair and the region seems useless i.e. \n",
    "               y_true[iframe,igridx,igridy,4] = 0 \"and\" the predicted region has no object that has IoU > 0.6\n",
    "               \n",
    "    conf_mask[iframe, igridy, igridx, ianchor] =  NO_OBJECT_SCALE\n",
    "               when there is no object assigned in (grid cell, anchor) pair but region seems to include some object\n",
    "               y_true[iframe,igridx,igridy,4] = 0 \"and\" the predicted region has some object that has IoU > 0.6\n",
    "               \n",
    "    conf_mask[iframe, igridy, igridx, ianchor] =  OBJECT_SCALE\n",
    "              when there is an object in (grid cell, anchor) pair        \n",
    "    '''\n",
    "\n",
    "    conf_mask = tf.to_float(best_ious < 0.6) * (1 - true_box_conf) * LAMBDA_NO_OBJECT\n",
    "    # penalize the confidence of the boxes, which are reponsible for corresponding ground truth box\n",
    "    conf_mask = conf_mask + true_box_conf_IOU * LAMBDA_OBJECT\n",
    "    return(conf_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment <code>get_conf_mask</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "input\n",
      "******************************\n",
      "best_ious         = Tensor(\"Max:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "true_box_conf     = Tensor(\"strided_slice_6:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "true_box_conf_IOU = Tensor(\"mul_9:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "LAMBDA_NO_OBJECT  = 1.0\n",
      "LAMBDA_OBJECT     = 5.0\n",
      "******************************\n",
      "output\n",
      "******************************\n",
      "conf_mask shape = (500, 13, 13, 4)\n"
     ]
    }
   ],
   "source": [
    "conf_mask_tf = get_conf_mask(best_ious_tf, \n",
    "                             true_box_conf_tf, \n",
    "                             true_box_conf_IOU_tf,\n",
    "                             LAMBDA_NO_OBJECT, \n",
    "                             LAMBDA_OBJECT)\n",
    "print(\"*\"*30 + \"\\ninput\\n\" + \"*\"*30)  \n",
    "print(\"best_ious         = {}\".format(best_ious_tf)) \n",
    "print(\"true_box_conf     = {}\".format(true_box_conf_tf))\n",
    "print(\"true_box_conf_IOU = {}\".format(true_box_conf_IOU_tf))\n",
    "print(\"LAMBDA_NO_OBJECT  = {}\".format(LAMBDA_NO_OBJECT)) \n",
    "print(\"LAMBDA_OBJECT     = {}\".format(LAMBDA_OBJECT))\n",
    "with tf.Session() as sess:\n",
    "    conf_mask = sess.run(conf_mask_tf)\n",
    "print(\"*\"*30 + \"\\noutput\\n\" + \"*\"*30)      \n",
    "print(\"conf_mask shape = {}\".format(conf_mask.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8: Calculate loss for the confidence $\\textrm{loss}_{i,j}^c$\n",
    "$$\n",
    "\\textrm{loss}_{i,j}^c =\n",
    "\\frac{1}{N^{conf}}\\left[\n",
    "\\lambda_{\\text{obj}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{obj}}\n",
    "\\left(\n",
    "IOU_{\\text{preduiction}_{i,j}}^{\\text{ground truth}_{i,j}} - \\widehat{C}_{i,j}\n",
    "\\right)^2+\n",
    "\\lambda_{\\textrm{noobj}}\n",
    "\\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B\n",
    "L_{i,j}^{\\text{noobj}}\n",
    "\\left(0 - \\widehat{C}_{i,j}\\right)\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "- $N^{conf} =  \\sum_{i=0}^{S^2}\n",
    "\\sum_{j=0}^B L_{i,j}^{\\text{obj}} + L_{i,j}^{\\text{noobj}}(1-L_{i,j}^{\\text{obj}})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_loss_conf(conf_mask,true_box_conf_IOU, pred_box_conf):  \n",
    "    '''\n",
    "    == input ==\n",
    "    \n",
    "    conf_mask         : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    true_box_conf_IOU : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    pred_box_conf     : tensor of shape (Nbatch, N grid h, N grid w, N anchor)\n",
    "    '''\n",
    "    # the number of (grid cell, anchor) pair that has an assigned object or\n",
    "    # that has no assigned object but some objects may be in bounding box.\n",
    "    # N conf\n",
    "    nb_conf_box  = tf.reduce_sum(tf.to_float(conf_mask  > 0.0))\n",
    "    loss_conf    = tf.reduce_sum(tf.square(true_box_conf_IOU-pred_box_conf) * conf_mask)  / (nb_conf_box  + 1e-6) / 2.\n",
    "    return(loss_conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment <code>calc_loss_conf</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "******************************\n",
      "input\n",
      "******************************\n",
      "conf_mask_tf         = Tensor(\"add_11:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "true_box_conf_IOU_tf = Tensor(\"mul_9:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "pred_box_conf_tf     = Tensor(\"Sigmoid_1:0\", shape=(500, 13, 13, 4), dtype=float32)\n",
      "******************************\n",
      "output\n",
      "******************************\n",
      "loss_conf = 0.1249\n"
     ]
    }
   ],
   "source": [
    "print(\"*\"*30 + \"\\ninput\\n\" + \"*\"*30)  \n",
    "print(\"conf_mask_tf         = {}\".format(conf_mask_tf)) \n",
    "print(\"true_box_conf_IOU_tf = {}\".format(true_box_conf_IOU_tf))\n",
    "print(\"pred_box_conf_tf     = {}\".format(pred_box_conf_tf))\n",
    "\n",
    "loss_conf_tf = calc_loss_conf(conf_mask_tf,true_box_conf_IOU_tf, pred_box_conf_tf)\n",
    "with tf.Session() as sess:\n",
    "    loss_conf = sess.run(loss_conf_tf) \n",
    "    \n",
    "print(\"*\"*30 + \"\\noutput\\n\" + \"*\"*30)      \n",
    "print(\"loss_conf = {:5.4f}\".format(loss_conf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <code>custom_loss(y_true,y_pred)</code>\n",
    "Finally combine all the calculation above into <code>custom_loss(y_true,y_pred)</code>\n",
    "Notice that true_boxes are tensor defined when the Keras model is declared. \n",
    "However, this tensor will not be passed as explicit arguments of the loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def custom_loss(y_true, y_pred):\n",
    "    '''\n",
    "    y_true : (N batch, N grid h, N grid w, N anchor, 4 + 1 + N classes)\n",
    "    y_true[irow, i_gridh, i_gridw, i_anchor, :4] = center_x, center_y, w, h\n",
    "    \n",
    "        center_x : The x coordinate center of the bounding box.\n",
    "                   Rescaled to range between 0 and N gird  w (e.g., ranging between [0,13)\n",
    "        center_y : The y coordinate center of the bounding box.\n",
    "                   Rescaled to range between 0 and N gird  h (e.g., ranging between [0,13)\n",
    "        w        : The width of the bounding box.\n",
    "                   Rescaled to range between 0 and N gird  w (e.g., ranging between [0,13)\n",
    "        h        : The height of the bounding box.\n",
    "                   Rescaled to range between 0 and N gird  h (e.g., ranging between [0,13)\n",
    "                   \n",
    "    y_true[irow, i_gridh, i_gridw, i_anchor, 4] = ground truth confidence\n",
    "        \n",
    "        ground truth confidence is 1 if object exists in this (anchor box, gird cell) pair\n",
    "    \n",
    "    y_true[irow, i_gridh, i_gridw, i_anchor, 5 + iclass] = 1 if the object is in category <iclass> else 0\n",
    "        \n",
    "    '''\n",
    "    total_recall = tf.Variable(0.)\n",
    "    \n",
    "    # Step 1: Adjust prediction output\n",
    "    cell_grid   = get_cell_grid(GRID_W,GRID_H,BATCH_SIZE,BOX)\n",
    "    pred_box_xy, pred_box_wh, pred_box_conf, pred_box_class = adjust_scale_prediction(y_pred,cell_grid,ANCHORS)\n",
    "    # Step 2: Extract ground truth output\n",
    "    true_box_xy, true_box_wh, true_box_conf, true_box_class = extract_ground_truth(y_true)\n",
    "    # Step 3: Calculate loss for the bounding box parameters\n",
    "    loss_xywh, coord_mask = calc_loss_xywh(true_box_conf,LAMBDA_COORD,\n",
    "                                           true_box_xy, pred_box_xy,true_box_wh,pred_box_wh)\n",
    "    # Step 4: Calculate loss for the class probabilities\n",
    "    loss_class  = calc_loss_class(true_box_conf,LAMBDA_CLASS,\n",
    "                                   true_box_class,pred_box_class)\n",
    "    # Step 5: For each (grid cell, anchor) pair, \n",
    "    #         calculate the IoU between predicted and ground truth bounding box\n",
    "    true_box_conf_IOU = calc_IOU_pred_true_assigned(true_box_conf,\n",
    "                                                    true_box_xy, true_box_wh,\n",
    "                                                    pred_box_xy, pred_box_wh)\n",
    "    # Step 6: For each predicted bounded box from (grid cell, anchor box), \n",
    "    #         calculate the best IOU, regardless of the ground truth anchor box that each object gets assigned.\n",
    "    best_ious = calc_IOU_pred_true_best(pred_box_xy,pred_box_wh,true_boxes)\n",
    "    # Step 7: For each grid cell, calculate the L_{i,j}^{noobj}\n",
    "    conf_mask = get_conf_mask(best_ious, true_box_conf, true_box_conf_IOU,LAMBDA_NO_OBJECT, LAMBDA_OBJECT)\n",
    "    # Step 8: Calculate loss for the confidence\n",
    "    loss_conf = calc_loss_conf(conf_mask,true_box_conf_IOU, pred_box_conf)\n",
    "\n",
    "    \n",
    "    loss = loss_xywh + loss_conf + loss_class\n",
    "    \n",
    "\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment <code>custom_loss</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.406112"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_boxes = tf.Variable(np.zeros_like(b_batch),dtype=\"float32\")\n",
    "loss_tf    = custom_loss(y_batch_tf, y_pred_tf) \n",
    "with tf.Session() as sess:\n",
    "    loss = sess.run(loss_tf,\n",
    "                    feed_dict = {true_boxes: b_batch})\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[FairyOnIce/ObjectDetectionYolo](https://github.com/FairyOnIce/ObjectDetectionYolo)\n",
    " contains this ipython notebook and all the functions that I defined in this notebook. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
